capture stdout and stderr

capture stdout and stderr

[Thomas Danckaert]

Hi Guilers,

I want to run an external process, and capture both its stdout and 
stderr output streams.  I didn't find an easy way to get a port for 
an external process' stderr, other than wrapping the call to the 
process in "with-error-to-port".

My questions are:
  0) Did I miss something obvious? :)
  1) why doesn't the second approach in the code below work (the 
output-string is always empty)?
  2) are there any other (better) solutions?
  3) I suppose my approach with (pipe) might block if the process 
writes too much data to stderr before I read from the pipe. Should I 
use "select" to interleave reads from stdout and stderr?

thank you!

Thomas

(use-modules (ice-9 format)
	     (ice-9 popen)
	     (ice-9 textual-ports))

(define (run-command)
   (let ((process (open-input-pipe "./stdoutstderr.sh")))
     (format #t "stdout: '~a'~%" (get-string-all process))
     (close-pipe process)))

;; Works:
(let ((err-pipe (pipe)))
   (with-error-to-port (cdr err-pipe) run-command)

   (close-port (cdr err-pipe))
   (format #t "stderr: '~a' ~%" (get-string-all (car err-pipe))))

;; Doesn't work: (get-output-string) returns empty string.
(let ((err-port (open-output-string)))
   (with-error-to-port err-port run-command)

   (format #t "stderr: '~a'~%" (get-output-string err-port))
   (close-output-port err-port))

Re: capture stdout and stderr

[Catonano]

2018-05-24 10:10 GMT+02:00 Thomas Danckaert <post@thomasdanckaert.be>:

> Hi Guilers,
>
> I want to run an external process, and capture both its stdout and stderr
> output streams.  I didn't find an easy way to get a port for an external
> process' stderr, other than wrapping the call to the process in
> "with-error-to-port".
>
> My questions are:
>  0) Did I miss something obvious? :)
>  1) why doesn't the second approach in the code below work (the
> output-string is always empty)?
>  2) are there any other (better) solutions?
>  3) I suppose my approach with (pipe) might block if the process writes
> too much data to stderr before I read from the pipe. Should I use "select"
> to interleave reads from stdout and stderr?
>
> thank you!
>
> Thomas
>
> (use-modules (ice-9 format)
>              (ice-9 popen)
>              (ice-9 textual-ports))
>
> (define (run-command)
>   (let ((process (open-input-pipe "./stdoutstderr.sh")))
>     (format #t "stdout: '~a'~%" (get-string-all process))
>     (close-pipe process)))
>
> ;; Works:
> (let ((err-pipe (pipe)))
>   (with-error-to-port (cdr err-pipe) run-command)
>
>   (close-port (cdr err-pipe))
>   (format #t "stderr: '~a' ~%" (get-string-all (car err-pipe))))
>
> ;; Doesn't work: (get-output-string) returns empty string.
> (let ((err-port (open-output-string)))
>   (with-error-to-port err-port run-command)
>
>   (format #t "stderr: '~a'~%" (get-output-string err-port))
>   (close-output-port err-port))
>
>

Hi Thomas

I'm the most incompetent guile user on earth and yet I want to venture into
offering a couple of observations

The first observation is that in the first case, you close the port
cotaining the error _prior_ to attempting reading from the port

In the second case, you attempt reading from the port AND THEN you close
the port

I undertsand that the writing on ports is mediated by some form of
buffering, or something, and closing the port also "flushes" the port,
meaning that if there are some remnants to be written yet, they get written

I don't even know of with-error-to-port automatically closes the port so
that manually closing it shouldn't be necessary

If with-error-to-port closes the port, then my observations doesn't apply

I run into this misundertsanding not a long time ago and I wrote about it
on Mastodon, here
https://mastodon.social/@catonano/99979214792830581

The second observation is that you could try to see forr yourself by
expanding your calls and visually inspect the expanded versions

They should differ in that you should be able to see the closing of the
port (I never checed this out)

Ok I rambled enough

Hope this helps !

Ciao

Re: capture stdout and stderr

[Thomas Danckaert]

From: Catonano <catonano@gmail.com>
Subject: Re: capture stdout and stderr
Date: Sat, 26 May 2018 12:24:11 +0200

> The first observation is that in the first case, you close the port 
> cotaining
> the error _prior_ to attempting reading from the port
>
> In the second case, you attempt reading from the port AND THEN you 
> close the
> port

Yes, I also found that a bit surprising when I wrote that.  However, 
with a string-port, you have to call get-output-string *before* 
closing the port, so that's why it's written like that 
(https://www.gnu.org/software/guile/manual/html_node/String-Ports.html).

> I undertsand that the writing on ports is mediated by some form of 
> buffering,
> or something, and closing the port also "flushes" the port, meaning 
> that if
> there are some remnants to be written yet, they get written
>
> I don't even know of with-error-to-port automatically closes the 
> port so that
> manually closing it shouldn't be necessary

My reason to close the port was in order to have an EOF at the end of 
the error output, so I can simply use get-string-all to retrieve it.  
 But yes, otherwise buffering might also come into play.  I think I 
checked that with-error-to-port didn't close the port for me.  I'm 
now leaving on a short vacation, so don't have time to check again in 
more detail, but it's a good suggestion.

Thanks!

Thomas

Re: capture stdout and stderr

[Thomas Danckaert]

From: Stefan Israelsson Tampe <stefan.itampe@gmail.com>
Subject: Re: capture stdout and stderr
Date: Thu, 24 May 2018 15:25:51 +0200

> I think that your code whould work but doesn't because string ports 
> are not file ports and system commands pipes out to file ports.
>
> 'open-input-pipe' uses 'scm_open_process' that returns only a 
> read-port and a write-port, the write port is then taped the output 
> string in your run-command.
> and it works.
>
> ALso when you create a err-port with pipe, you create a file port 
> nad this will be used translated as an error port for the system 
> comand and it all works.
> However the failing program uses a string port which is not a file 
> port and it does not end up capturing the error output in the 
> system command.
>
> One would wish that guile if the error port is not a file port make 
> a file port, as done with 'pipe' and create a thread that pipes the 
> data from the file port to the non file port and your code
> would work. Also similarly behavior for read and write ports 
> towards the shell.
>
> the relevant code is in posix.c, scm_open_process

Hi! Thanks for the message.  Indeed, that makes sense.  In fact I had 
some vague intuition that some magic would be needed to channel the 
stderr output into such a string port.  I think the manual could be 
more explicit about that, though? (e.g. in the section on  default 
ports 
https://www.gnu.org/software/guile/manual/html_node/Default-Ports.html#Default-Ports 
).  Maybe I should propose a patch to the docs, stating that those 
"with-x-to-port" methods only work on file ports.

Thomas

Re: capture stdout and stderr

[Catonano]

2018-05-27 19:26 GMT+02:00 Thomas Danckaert <post@thomasdanckaert.be>:

> From: Stefan Israelsson Tampe <stefan.itampe@gmail.com>
> Subject: Re: capture stdout and stderr
> Date: Thu, 24 May 2018 15:25:51 +0200
>
> I think that your code whould work but doesn't because string ports are
>> not file ports and system commands pipes out to file ports.
>>
>> 'open-input-pipe' uses 'scm_open_process' that returns only a read-port
>> and a write-port, the write port is then taped the output string in your
>> run-command.
>> and it works.
>>
>> ALso when you create a err-port with pipe, you create a file port nad
>> this will be used translated as an error port for the system comand and it
>> all works.
>> However the failing program uses a string port which is not a file port
>> and it does not end up capturing the error output in the system command.
>>
>> One would wish that guile if the error port is not a file port make a
>> file port, as done with 'pipe' and create a thread that pipes the data from
>> the file port to the non file port and your code
>> would work. Also similarly behavior for read and write ports towards the
>> shell.
>>
>> the relevant code is in posix.c, scm_open_process
>>
>
> Hi! Thanks for the message.  Indeed, that makes sense.  In fact I had some
> vague intuition that some magic would be needed to channel the stderr
> output into such a string port.  I think the manual could be more explicit
> about that, though? (e.g. in the section on  default ports
> https://www.gnu.org/software/guile/manual/html_node/Default-
> Ports.html#Default-Ports ).  Maybe I should propose a patch to the docs,
> stating that those "with-x-to-port" methods only work on file ports.
>
> Thomas
>
>
>
Thomas

thank you for sharing the solution