* Forwarding stdout/stderr of a child process in batch mode @ 2020-12-06 13:30 Paul Pogonyshev 2020-12-06 13:36 ` Zhu Zihao 0 siblings, 1 reply; 6+ messages in thread From: Paul Pogonyshev @ 2020-12-06 13:30 UTC (permalink / raw) To: Emacs developers Hi, I'm using Emacs in batch mode. I need to invoke a child process that is a longish operation (a few minutes). During this time, it writes to its stdout, so user will see that it is working and what exactly is being done. However, if I invoke it from Emacs (e.g. using `call-process') I see no way of forwarding this output to the "real" stdout. So, for a user this looks like the process (or batch Emacs on top of it) is hung. Am I missing a way to forward output? Paul ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode 2020-12-06 13:30 Forwarding stdout/stderr of a child process in batch mode Paul Pogonyshev @ 2020-12-06 13:36 ` Zhu Zihao 2020-12-06 21:40 ` Paul Pogonyshev 0 siblings, 1 reply; 6+ messages in thread From: Zhu Zihao @ 2020-12-06 13:36 UTC (permalink / raw) To: Paul Pogonyshev; +Cc: emacs-devel [-- Attachment #1: Type: text/plain, Size: 805 bytes --] IMO, in batch mode, `message` writes to stderr, `princ` writes to stdout. You can install a filter for childprocess, and run functions I mentioned above to forward these outputs. Paul Pogonyshev writes: > Hi, > > I'm using Emacs in batch mode. I need to invoke a child process that > is a longish operation (a few minutes). During this time, it writes to > its stdout, so user will see that it is working and what exactly is > being done. However, if I invoke it from Emacs (e.g. using > `call-process') I see no way of forwarding this output to the "real" > stdout. So, for a user this looks like the process (or batch Emacs on > top of it) is hung. > > Am I missing a way to forward output? > > Paul -- Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp Zihao [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 515 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode 2020-12-06 13:36 ` Zhu Zihao @ 2020-12-06 21:40 ` Paul Pogonyshev 2020-12-08 15:55 ` Paul Pogonyshev 0 siblings, 1 reply; 6+ messages in thread From: Paul Pogonyshev @ 2020-12-06 21:40 UTC (permalink / raw) To: Zhu Zihao; +Cc: Emacs developers Thank you, this seems to work (haven't tried separating stderr from stdout yet). Is there a normal way to wait for asynchronous process to terminate? Currently I have managed to make it work by throwing from sentinel, but this feels a bit hackish, maybe there is something better? Currently I have something like this: (let ((process (make-process ... :sentinel (lambda (_process _event) (throw 'done nil))))) (catch 'done (while (process-live-p process) (accept-process-output nil 60))) ... Paul On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote: > > > IMO, in batch mode, `message` writes to stderr, `princ` writes to > stdout. You can install a filter for childprocess, and run functions I > mentioned above to forward these outputs. > > > Paul Pogonyshev writes: > > > Hi, > > > > I'm using Emacs in batch mode. I need to invoke a child process that > > is a longish operation (a few minutes). During this time, it writes to > > its stdout, so user will see that it is working and what exactly is > > being done. However, if I invoke it from Emacs (e.g. using > > `call-process') I see no way of forwarding this output to the "real" > > stdout. So, for a user this looks like the process (or batch Emacs on > > top of it) is hung. > > > > Am I missing a way to forward output? > > > > Paul > > > -- > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp > > Zihao ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode 2020-12-06 21:40 ` Paul Pogonyshev @ 2020-12-08 15:55 ` Paul Pogonyshev 2020-12-08 21:43 ` Bruno Barbier 0 siblings, 1 reply; 6+ messages in thread From: Paul Pogonyshev @ 2020-12-08 15:55 UTC (permalink / raw) To: Zhu Zihao; +Cc: Emacs developers [-- Attachment #1: Type: text/plain, Size: 3320 bytes --] So, I got to separating stdout and stderr, but I get them in a wrong order now... Here is an example Python program that I use to generate output (store as `10.py'): #! /usr/bin/env python import sys import time for k in range (10): sys.stdout.write ('line %d to stdout\n' % k) sys.stderr.write ('line %d to stderr\n' % k) time.sleep (0.2) And the Emacs script (store as `fwd.el') to be executed as $ emacs --batch --script fwd.el that gives output in a wrong (compared to simply executing `./10.py' from console, including M-x shell or M-x eshell) order: (let ((process (make-process :name "10" :command `(,(expand-file-name "10.py")) :filter (lambda (_process string) (princ string)) :stderr (make-pipe-process :name "10/stderr" :filter (lambda (_process string) (message "%s" string))) :sentinel (lambda (&rest _ignored) (throw 'interrupted nil))))) (while (process-live-p process) (catch 'interrupted (accept-process-output nil 60)))) When I run the Emacs script, I receive all stderr output line-by-line (which is fine), but all stdout output comes in one chunk at the very end, which is not... I cannot really use `pty' connection type, because I want (in real usecase, not in the script) to additionally keep stdout and stderr in separate buffers, for possible later processing. Also, with pty I wouldn't be able to use `message' or `princ' depending on the stream used by the child process. Is there any way to fix this? Paul On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> wrote: > Thank you, this seems to work (haven't tried separating stderr from > stdout yet). Is there a normal way to wait for asynchronous process to > terminate? Currently I have managed to make it work by throwing from > sentinel, but this feels a bit hackish, maybe there is something > better? > > Currently I have something like this: > > (let ((process (make-process ... :sentinel (lambda (_process > _event) (throw 'done nil))))) > (catch 'done > (while (process-live-p process) > (accept-process-output nil 60))) > ... > > Paul > > On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote: > > > > > > IMO, in batch mode, `message` writes to stderr, `princ` writes to > > stdout. You can install a filter for childprocess, and run functions I > > mentioned above to forward these outputs. > > > > > > Paul Pogonyshev writes: > > > > > Hi, > > > > > > I'm using Emacs in batch mode. I need to invoke a child process that > > > is a longish operation (a few minutes). During this time, it writes to > > > its stdout, so user will see that it is working and what exactly is > > > being done. However, if I invoke it from Emacs (e.g. using > > > `call-process') I see no way of forwarding this output to the "real" > > > stdout. So, for a user this looks like the process (or batch Emacs on > > > top of it) is hung. > > > > > > Am I missing a way to forward output? > > > > > > Paul > > > > > > -- > > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp > > > > Zihao > [-- Attachment #2: Type: text/html, Size: 4429 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode 2020-12-08 15:55 ` Paul Pogonyshev @ 2020-12-08 21:43 ` Bruno Barbier 2020-12-09 1:14 ` Paul Pogonyshev 0 siblings, 1 reply; 6+ messages in thread From: Bruno Barbier @ 2020-12-08 21:43 UTC (permalink / raw) To: Paul Pogonyshev, Zhu Zihao; +Cc: Emacs developers Hi Paul, Paul Pogonyshev <pogonyshev@gmail.com> writes: > So, I got to separating stdout and stderr, but I get them in a wrong > order now... Here is an example Python program that I use to generate > output (store as `10.py'): ... > When I run the Emacs script, I receive all stderr output line-by-line > (which is fine), but all stdout output comes in one chunk at the very > end, which is not... > You may want to pass the "-u" option to Python, so that it uses unbuffered outputs. You're example is working for me, when using the "-u" option; i.e. I'm getting interleaved chunks of stderr/stdout. Bruno > I cannot really use `pty' connection type, because I want (in real > usecase, not in the script) to additionally keep stdout and stderr in > separate buffers, for possible later processing. Also, with pty I > wouldn't be able to use `message' or `princ' depending on the stream > used by the child process. > > Is there any way to fix this? > > Paul > > > On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> wrote: > >> Thank you, this seems to work (haven't tried separating stderr from >> stdout yet). Is there a normal way to wait for asynchronous process to >> terminate? Currently I have managed to make it work by throwing from >> sentinel, but this feels a bit hackish, maybe there is something >> better? >> >> Currently I have something like this: >> >> (let ((process (make-process ... :sentinel (lambda (_process >> _event) (throw 'done nil))))) >> (catch 'done >> (while (process-live-p process) >> (accept-process-output nil 60))) >> ... >> >> Paul >> >> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote: >> > >> > >> > IMO, in batch mode, `message` writes to stderr, `princ` writes to >> > stdout. You can install a filter for childprocess, and run functions I >> > mentioned above to forward these outputs. >> > >> > >> > Paul Pogonyshev writes: >> > >> > > Hi, >> > > >> > > I'm using Emacs in batch mode. I need to invoke a child process that >> > > is a longish operation (a few minutes). During this time, it writes to >> > > its stdout, so user will see that it is working and what exactly is >> > > being done. However, if I invoke it from Emacs (e.g. using >> > > `call-process') I see no way of forwarding this output to the "real" >> > > stdout. So, for a user this looks like the process (or batch Emacs on >> > > top of it) is hung. >> > > >> > > Am I missing a way to forward output? >> > > >> > > Paul >> > >> > >> > -- >> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp >> > >> > Zihao >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Forwarding stdout/stderr of a child process in batch mode 2020-12-08 21:43 ` Bruno Barbier @ 2020-12-09 1:14 ` Paul Pogonyshev 0 siblings, 0 replies; 6+ messages in thread From: Paul Pogonyshev @ 2020-12-09 1:14 UTC (permalink / raw) To: Bruno Barbier; +Cc: Zhu Zihao, Emacs developers [-- Attachment #1: Type: text/plain, Size: 3848 bytes --] > You may want to pass the "-u" option to Python, so that it uses > unbuffered outputs. > > You're example is working for me, when using the "-u" option; i.e. I'm > getting interleaved chunks of stderr/stdout. It is only an example. In real usecase I will not call a Python script, in fact, it can be any executable whatsoever, I don't know in advance. Also, shell itself doesn't need `-u'. For example: sh -c 'sh -c "python 10.py"' sh -c 'sh -c "python 10.py"' >/dev/null The second command shows that it doesn't write everything to stdout, it still maintains stdout/stderr separation. The nested `-c' shows that it works through any (I guess) level of indirection, and also if I use different shells (`bash', `dash'), so it doesn't look like shell is special- casing, it really can handle any commands. Finally, even with `python -u' I get non-ideal behavior from Emacs: stderr chunks come before the stdout one, even though the script writes to stdout before stderr. Paul On Tue, 8 Dec 2020 at 22:43, Bruno Barbier <brubar.cs@gmail.com> wrote: > > Hi Paul, > > Paul Pogonyshev <pogonyshev@gmail.com> writes: > > > So, I got to separating stdout and stderr, but I get them in a wrong > > order now... Here is an example Python program that I use to generate > > output (store as `10.py'): > ... > > > When I run the Emacs script, I receive all stderr output line-by-line > > (which is fine), but all stdout output comes in one chunk at the very > > end, which is not... > > > > You may want to pass the "-u" option to Python, so that it uses > unbuffered outputs. > > You're example is working for me, when using the "-u" option; i.e. I'm > getting interleaved chunks of stderr/stdout. > > Bruno > > > I cannot really use `pty' connection type, because I want (in real > > usecase, not in the script) to additionally keep stdout and stderr in > > separate buffers, for possible later processing. Also, with pty I > > wouldn't be able to use `message' or `princ' depending on the stream > > used by the child process. > > > > Is there any way to fix this? > > > > Paul > > > > > > On Sun, 6 Dec 2020 at 22:40, Paul Pogonyshev <pogonyshev@gmail.com> > wrote: > > > >> Thank you, this seems to work (haven't tried separating stderr from > >> stdout yet). Is there a normal way to wait for asynchronous process to > >> terminate? Currently I have managed to make it work by throwing from > >> sentinel, but this feels a bit hackish, maybe there is something > >> better? > >> > >> Currently I have something like this: > >> > >> (let ((process (make-process ... :sentinel (lambda (_process > >> _event) (throw 'done nil))))) > >> (catch 'done > >> (while (process-live-p process) > >> (accept-process-output nil 60))) > >> ... > >> > >> Paul > >> > >> On Sun, 6 Dec 2020 at 14:37, Zhu Zihao <all_but_last@163.com> wrote: > >> > > >> > > >> > IMO, in batch mode, `message` writes to stderr, `princ` writes to > >> > stdout. You can install a filter for childprocess, and run functions I > >> > mentioned above to forward these outputs. > >> > > >> > > >> > Paul Pogonyshev writes: > >> > > >> > > Hi, > >> > > > >> > > I'm using Emacs in batch mode. I need to invoke a child process that > >> > > is a longish operation (a few minutes). During this time, it writes > to > >> > > its stdout, so user will see that it is working and what exactly is > >> > > being done. However, if I invoke it from Emacs (e.g. using > >> > > `call-process') I see no way of forwarding this output to the "real" > >> > > stdout. So, for a user this looks like the process (or batch Emacs > on > >> > > top of it) is hung. > >> > > > >> > > Am I missing a way to forward output? > >> > > > >> > > Paul > >> > > >> > > >> > -- > >> > Retrieve my PGP public key: https://meta.sr.ht/~citreu.pgp > >> > > >> > Zihao > >> > [-- Attachment #2: Type: text/html, Size: 5823 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-12-09 1:14 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-12-06 13:30 Forwarding stdout/stderr of a child process in batch mode Paul Pogonyshev 2020-12-06 13:36 ` Zhu Zihao 2020-12-06 21:40 ` Paul Pogonyshev 2020-12-08 15:55 ` Paul Pogonyshev 2020-12-08 21:43 ` Bruno Barbier 2020-12-09 1:14 ` Paul Pogonyshev
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).