* Forwarding stdout/stderr of a child process in batch mode
@ 2020-12-06 13:30 Paul Pogonyshev
2020-12-06 13:36 ` Zhu Zihao
0 siblings, 1 reply; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-06 13:30 UTC (permalink / raw)
To: Emacs developers
Hi,
I'm using Emacs in batch mode. I need to invoke a child process that
is a longish operation (a few minutes). During this time, it writes to
its stdout, so user will see that it is working and what exactly is
being done. However, if I invoke it from Emacs (e.g. using
`call-process') I see no way of forwarding this output to the "real"
stdout. So, for a user this looks like the process (or batch Emacs on
top of it) is hung.
Am I missing a way to forward output?
Paul
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode
2020-12-06 13:30 Forwarding stdout/stderr of a child process in batch mode Paul Pogonyshev
@ 2020-12-06 13:36 ` Zhu Zihao
2020-12-06 21:40 ` Paul Pogonyshev
0 siblings, 1 reply; 6+ messages in thread
From: Zhu Zihao @ 2020-12-06 13:36 UTC (permalink / raw)
To: Paul Pogonyshev; +Cc: emacs-devel
[-- Attachment #1: Type: text/plain, Size: 805 bytes --]
IMO, in batch mode, `message` writes to stderr, `princ` writes to
stdout. You can install a filter for childprocess, and run functions I
mentioned above to forward these outputs.
Paul Pogonyshev writes:
> Hi,
>
> I'm using Emacs in batch mode. I need to invoke a child process that
> is a longish operation (a few minutes). During this time, it writes to
> its stdout, so user will see that it is working and what exactly is
> being done. However, if I invoke it from Emacs (e.g. using
> `call-process') I see no way of forwarding this output to the "real"
> stdout. So, for a user this looks like the process (or batch Emacs on
> top of it) is hung.
>
> Am I missing a way to forward output?
>
> Paul
--
Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
Zihao
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 515 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode
2020-12-06 13:36 ` Zhu Zihao
@ 2020-12-06 21:40 ` Paul Pogonyshev
2020-12-08 15:55 ` Paul Pogonyshev
0 siblings, 1 reply; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-06 21:40 UTC (permalink / raw)
To: Zhu Zihao; +Cc: Emacs developers
Thank you, this seems to work (haven't tried separating stderr from
stdout yet). Is there a normal way to wait for asynchronous process to
terminate? Currently I have managed to make it work by throwing from
sentinel, but this feels a bit hackish, maybe there is something
better?
Currently I have something like this:
(let ((process (make-process ... :sentinel (lambda (_process
_event) (throw 'done nil)))))
(catch 'done
(while (process-live-p process)
(accept-process-output nil 60)))
...
Paul
On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
>
>
> IMO, in batch mode, `message` writes to stderr, `princ` writes to
> stdout. You can install a filter for childprocess, and run functions I
> mentioned above to forward these outputs.
>
>
> Paul Pogonyshev writes:
>
> > Hi,
> >
> > I'm using Emacs in batch mode. I need to invoke a child process that
> > is a longish operation (a few minutes). During this time, it writes to
> > its stdout, so user will see that it is working and what exactly is
> > being done. However, if I invoke it from Emacs (e.g. using
> > `call-process') I see no way of forwarding this output to the "real"
> > stdout. So, for a user this looks like the process (or batch Emacs on
> > top of it) is hung.
> >
> > Am I missing a way to forward output?
> >
> > Paul
>
>
> --
> Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
>
> Zihao
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode
2020-12-06 21:40 ` Paul Pogonyshev
@ 2020-12-08 15:55 ` Paul Pogonyshev
2020-12-08 21:43 ` Bruno Barbier
0 siblings, 1 reply; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-08 15:55 UTC (permalink / raw)
To: Zhu Zihao; +Cc: Emacs developers
[-- Attachment #1: Type: text/plain, Size: 3320 bytes --]
So, I got to separating stdout and stderr, but I get them in a wrong
order now... Here is an example Python program that I use to generate
output (store as `10.py'):
#! /usr/bin/env python
import sys
import time
for k in range (10):
sys.stdout.write ('line %d to stdout\n' % k)
sys.stderr.write ('line %d to stderr\n' % k)
time.sleep (0.2)
And the Emacs script (store as `fwd.el') to be executed as
$ emacs --batch --script fwd.el
that gives output in a wrong (compared to simply executing `./10.py'
from console, including M-x shell or M-x eshell) order:
(let ((process (make-process :name "10"
:command `(,(expand-file-name "10.py"))
:filter (lambda (_process string) (princ
string))
:stderr (make-pipe-process :name
"10/stderr"
:filter
(lambda (_process string) (message "%s" string)))
:sentinel (lambda (&rest _ignored) (throw
'interrupted nil)))))
(while (process-live-p process)
(catch 'interrupted
(accept-process-output nil 60))))
When I run the Emacs script, I receive all stderr output line-by-line
(which is fine), but all stdout output comes in one chunk at the very
end, which is not...
I cannot really use `pty' connection type, because I want (in real
usecase, not in the script) to additionally keep stdout and stderr in
separate buffers, for possible later processing. Also, with pty I
wouldn't be able to use `message' or `princ' depending on the stream
used by the child process.
Is there any way to fix this?
Paul
On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> wrote:
> Thank you, this seems to work (haven't tried separating stderr from
> stdout yet). Is there a normal way to wait for asynchronous process to
> terminate? Currently I have managed to make it work by throwing from
> sentinel, but this feels a bit hackish, maybe there is something
> better?
>
> Currently I have something like this:
>
> (let ((process (make-process ... :sentinel (lambda (_process
> _event) (throw 'done nil)))))
> (catch 'done
> (while (process-live-p process)
> (accept-process-output nil 60)))
> ...
>
> Paul
>
> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
> >
> >
> > IMO, in batch mode, `message` writes to stderr, `princ` writes to
> > stdout. You can install a filter for childprocess, and run functions I
> > mentioned above to forward these outputs.
> >
> >
> > Paul Pogonyshev writes:
> >
> > > Hi,
> > >
> > > I'm using Emacs in batch mode. I need to invoke a child process that
> > > is a longish operation (a few minutes). During this time, it writes to
> > > its stdout, so user will see that it is working and what exactly is
> > > being done. However, if I invoke it from Emacs (e.g. using
> > > `call-process') I see no way of forwarding this output to the "real"
> > > stdout. So, for a user this looks like the process (or batch Emacs on
> > > top of it) is hung.
> > >
> > > Am I missing a way to forward output?
> > >
> > > Paul
> >
> >
> > --
> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
> >
> > Zihao
>
[-- Attachment #2: Type: text/html, Size: 4429 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode
2020-12-08 15:55 ` Paul Pogonyshev
@ 2020-12-08 21:43 ` Bruno Barbier
2020-12-09 1:14 ` Paul Pogonyshev
0 siblings, 1 reply; 6+ messages in thread
From: Bruno Barbier @ 2020-12-08 21:43 UTC (permalink / raw)
To: Paul Pogonyshev, Zhu Zihao; +Cc: Emacs developers
Hi Paul,
Paul Pogonyshev <pogonyshev@gmail.com> writes:
> So, I got to separating stdout and stderr, but I get them in a wrong
> order now... Here is an example Python program that I use to generate
> output (store as `10.py'):
...
> When I run the Emacs script, I receive all stderr output line-by-line
> (which is fine), but all stdout output comes in one chunk at the very
> end, which is not...
>
You may want to pass the "-u" option to Python, so that it uses
unbuffered outputs.
You're example is working for me, when using the "-u" option; i.e. I'm
getting interleaved chunks of stderr/stdout.
Bruno
> I cannot really use `pty' connection type, because I want (in real
> usecase, not in the script) to additionally keep stdout and stderr in
> separate buffers, for possible later processing. Also, with pty I
> wouldn't be able to use `message' or `princ' depending on the stream
> used by the child process.
>
> Is there any way to fix this?
>
> Paul
>
>
> On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> wrote:
>
>> Thank you, this seems to work (haven't tried separating stderr from
>> stdout yet). Is there a normal way to wait for asynchronous process to
>> terminate? Currently I have managed to make it work by throwing from
>> sentinel, but this feels a bit hackish, maybe there is something
>> better?
>>
>> Currently I have something like this:
>>
>> (let ((process (make-process ... :sentinel (lambda (_process
>> _event) (throw 'done nil)))))
>> (catch 'done
>> (while (process-live-p process)
>> (accept-process-output nil 60)))
>> ...
>>
>> Paul
>>
>> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
>> >
>> >
>> > IMO, in batch mode, `message` writes to stderr, `princ` writes to
>> > stdout. You can install a filter for childprocess, and run functions I
>> > mentioned above to forward these outputs.
>> >
>> >
>> > Paul Pogonyshev writes:
>> >
>> > > Hi,
>> > >
>> > > I'm using Emacs in batch mode. I need to invoke a child process that
>> > > is a longish operation (a few minutes). During this time, it writes to
>> > > its stdout, so user will see that it is working and what exactly is
>> > > being done. However, if I invoke it from Emacs (e.g. using
>> > > `call-process') I see no way of forwarding this output to the "real"
>> > > stdout. So, for a user this looks like the process (or batch Emacs on
>> > > top of it) is hung.
>> > >
>> > > Am I missing a way to forward output?
>> > >
>> > > Paul
>> >
>> >
>> > --
>> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
>> >
>> > Zihao
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode
2020-12-08 21:43 ` Bruno Barbier
@ 2020-12-09 1:14 ` Paul Pogonyshev
0 siblings, 0 replies; 6+ messages in thread
From: Paul Pogonyshev @ 2020-12-09 1:14 UTC (permalink / raw)
To: Bruno Barbier; +Cc: Zhu Zihao, Emacs developers
[-- Attachment #1: Type: text/plain, Size: 3848 bytes --]
> You may want to pass the "-u" option to Python, so that it uses
> unbuffered outputs.
>
> You're example is working for me, when using the "-u" option; i.e. I'm
> getting interleaved chunks of stderr/stdout.
It is only an example. In real usecase I will not call a Python script, in
fact, it can be any executable whatsoever, I don't know in advance. Also,
shell itself doesn't need `-u'. For example:
sh -c 'sh -c "python 10.py"'
sh -c 'sh -c "python 10.py"' >/dev/null
The second command shows that it doesn't write everything to stdout,
it still maintains stdout/stderr separation. The nested `-c' shows that
it works through any (I guess) level of indirection, and also if I use
different shells (`bash', `dash'), so it doesn't look like shell is special-
casing, it really can handle any commands.
Finally, even with `python -u' I get non-ideal behavior from Emacs:
stderr chunks come before the stdout one, even though the script writes
to stdout before stderr.
Paul
On Tue, 8 Dec 2020 at 22:43, Bruno Barbier <brubar.cs@gmail.com> wrote:
>
> Hi Paul,
>
> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>
> > So, I got to separating stdout and stderr, but I get them in a wrong
> > order now... Here is an example Python program that I use to generate
> > output (store as `10.py'):
> ...
>
> > When I run the Emacs script, I receive all stderr output line-by-line
> > (which is fine), but all stdout output comes in one chunk at the very
> > end, which is not...
> >
>
> You may want to pass the "-u" option to Python, so that it uses
> unbuffered outputs.
>
> You're example is working for me, when using the "-u" option; i.e. I'm
> getting interleaved chunks of stderr/stdout.
>
> Bruno
>
> > I cannot really use `pty' connection type, because I want (in real
> > usecase, not in the script) to additionally keep stdout and stderr in
> > separate buffers, for possible later processing. Also, with pty I
> > wouldn't be able to use `message' or `princ' depending on the stream
> > used by the child process.
> >
> > Is there any way to fix this?
> >
> > Paul
> >
> >
> > On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com>
> wrote:
> >
> >> Thank you, this seems to work (haven't tried separating stderr from
> >> stdout yet). Is there a normal way to wait for asynchronous process to
> >> terminate? Currently I have managed to make it work by throwing from
> >> sentinel, but this feels a bit hackish, maybe there is something
> >> better?
> >>
> >> Currently I have something like this:
> >>
> >> (let ((process (make-process ... :sentinel (lambda (_process
> >> _event) (throw 'done nil)))))
> >> (catch 'done
> >> (while (process-live-p process)
> >> (accept-process-output nil 60)))
> >> ...
> >>
> >> Paul
> >>
> >> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote:
> >> >
> >> >
> >> > IMO, in batch mode, `message` writes to stderr, `princ` writes to
> >> > stdout. You can install a filter for childprocess, and run functions I
> >> > mentioned above to forward these outputs.
> >> >
> >> >
> >> > Paul Pogonyshev writes:
> >> >
> >> > > Hi,
> >> > >
> >> > > I'm using Emacs in batch mode. I need to invoke a child process that
> >> > > is a longish operation (a few minutes). During this time, it writes
> to
> >> > > its stdout, so user will see that it is working and what exactly is
> >> > > being done. However, if I invoke it from Emacs (e.g. using
> >> > > `call-process') I see no way of forwarding this output to the "real"
> >> > > stdout. So, for a user this looks like the process (or batch Emacs
> on
> >> > > top of it) is hung.
> >> > >
> >> > > Am I missing a way to forward output?
> >> > >
> >> > > Paul
> >> >
> >> >
> >> > --
> >> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp
> >> >
> >> > Zihao
> >>
>
[-- Attachment #2: Type: text/html, Size: 5823 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-12-09 1:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-06 13:30 Forwarding stdout/stderr of a child process in batch mode Paul Pogonyshev
2020-12-06 13:36 ` Zhu Zihao
2020-12-06 21:40 ` Paul Pogonyshev
2020-12-08 15:55 ` Paul Pogonyshev
2020-12-08 21:43 ` Bruno Barbier
2020-12-09 1:14 ` Paul Pogonyshev
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).