Subprocess API.

Subprocess API.

[Mathieu Othacehe]

Hi,

I recently used "open-pipe*" to launch a process but was unable to read
from stderr. This subject was already discussed on this ml here : 

https://lists.gnu.org/archive/html/guile-user/2015-04/msg00003.html

Racket seems to have procedures to provide stdout/stdin/stderr ports for
a given subprocess[1].

Mark, you said this subject was on your TODO list, is there anything
available or would it be possible to develop a racket like API ?

Thanks,

Mathieu

[1]:  https://docs.racket-lang.org/reference/subprocess.html

Re: Subprocess API.

[Chris Vine]

On Sat, 23 Sep 2017 11:58:34 +0200
Mathieu Othacehe <m.othacehe@gmail.com> wrote:
> Hi,
> 
> I recently used "open-pipe*" to launch a process but was unable to
> read from stderr. This subject was already discussed on this ml
> here : 
> 
> https://lists.gnu.org/archive/html/guile-user/2015-04/msg00003.html
> 
> Racket seems to have procedures to provide stdout/stdin/stderr ports
> for a given subprocess[1].
> 
> Mark, you said this subject was on your TODO list, is there anything
> available or would it be possible to develop a racket like API ?

The run-concurrently+ procedure in guile-lib
( http://www.nongnu.org/guile-lib/doc/ref/os.process/ ) should do what
you want.  Construct a pipe with guile's POSIX 'pipe' procedure and pass
the write port to the run-concurrently+ procedure for file descriptor 2
(stderr) and then read from the read port.  Close the write port on the
reader side before reading so that when the sub-process ends, the read
will return with an eof-object.

This works exactly as you would expect from its POSIX equivalents and
has the advantage that you can read from the pipe as the sub-process is
proceeding rather than just collect at the end.

Having said that it would be nice to have this in more formalized form
in guile itself, say something along the lines of ocaml's
Unix.create_process function.

Cjros

Re: Subprocess API.

[Mathieu Othacehe]

Hi Chris,

> This works exactly as you would expect from its POSIX equivalents and
> has the advantage that you can read from the pipe as the sub-process is
> proceeding rather than just collect at the end.

Thank you ! Following your suggestion, I ended-up with :

--8<---------------cut here---------------start------------->8---
(let* ((err-pipe (pipe))
       (out-pipe (pipe))
       (read-out (car out-pipe))
       (write-out (cdr out-pipe))
       (read-err (car err-pipe))
       (write-err (cdr err-pipe))
       (pid (run-concurrently+
             (apply tail-call-program "...")
             (write-out 1)
             (write-err 2)))
       (ret (status:exit-val (cdr (waitpid pid)))))
  (close-port write-out)
  (close-port write-err)
  (let ((output (read-string read-out))
        (error (read-string read-err)))
    (close-port read-out)
    (close-port read-err)
    (case ret
      ((0) output)
      (else (raise ...)))))
--8<---------------cut here---------------end--------------->8---

which seems to work. However, run-concurrently+ uses "primitive-fork"
which is forbiden in a multi-thread context (sadly, mine).

Do you have any idea on how to overcome this ?

Thanks,

Mathieu

Re: Subprocess API.

[Chris Vine]

On Mon, 25 Sep 2017 19:14:22 +0200
Mathieu Othacehe <m.othacehe@gmail.com> wrote:
> Hi Chris,
> 
> > This works exactly as you would expect from its POSIX equivalents
> > and has the advantage that you can read from the pipe as the
> > sub-process is proceeding rather than just collect at the end.  
> 
> Thank you ! Following your suggestion, I ended-up with :
> 
> --8<---------------cut here---------------start------------->8---
> (let* ((err-pipe (pipe))
>        (out-pipe (pipe))
>        (read-out (car out-pipe))
>        (write-out (cdr out-pipe))
>        (read-err (car err-pipe))
>        (write-err (cdr err-pipe))
>        (pid (run-concurrently+
>              (apply tail-call-program "...")
>              (write-out 1)
>              (write-err 2)))
>        (ret (status:exit-val (cdr (waitpid pid)))))
>   (close-port write-out)
>   (close-port write-err)
>   (let ((output (read-string read-out))
>         (error (read-string read-err)))
>     (close-port read-out)
>     (close-port read-errs 
>     (case ret
>       ((0) output)
>       (else (raise ...)))))
> --8<---------------cut here---------------end--------------->8---
> 
> which seems to work. However, run-concurrently+ uses "primitive-fork"
> which is forbiden in a multi-thread context (sadly, mine).
> 
> Do you have any idea on how to overcome this ?

Any launching of a new process requires a fork and if (as appears to
be your intention) you want to replace the process image with a new
one, an exec.  As you appear to know, POSIX allows only
async-signal-safe functions to be called in a multi-threaded program
between the fork and the exec, although glibc does relax this somewhat.

Since anything you do in guile between the fork and the exec has the
potential to allocate memory, that appears to mean that, as you say, you
cannot call primitive-fork in a guile program at a time when it is
running more than one thread.  If so, I do not know how to circumvent
that: you could consider launching the new process in C code via the
guile FFI so you can ensure that no non-async-signal-safe code is
called at the wrong time; but presumably you would still have by some
means to prevent the garbage collector from being able to start a
memory reclaiming run in the new process after the fork and before the
exec, and again I do not know how you would do that.  You would also
need to block system asyncs before forking (and unblock after the fork
in the original process) but that is trivial to do.

As regards your code, if you do not need to distinguish between stdout
and stderr, you would do better to have only one pipe and use the write
port of the pipe for both of descriptors 1 and 2.  That means that you
could read the pipe while the new process is proceeding rather than
after it has finished (which risks filling up the pipe): just loop
reading the read end of the pipe until an eof-object is received, and
then call waitpid after that.

Chris

Re: Subprocess API.

[Chris Vine]

On Mon, 25 Sep 2017 19:59:39 +0100
Chris Vine <vine35792468@gmail.com> wrote:
> ... you could consider launching the new process in C code via the
> guile FFI so you can ensure that no non-async-signal-safe code is
> called at the wrong time; but presumably you would still have by some
> means to prevent the garbage collector from being able to start a
> memory reclaiming run in the new process after the fork and before the
> exec, and again I do not know how you would do that.  You would also
> need to block system asyncs before forking (and unblock after the fork
> in the original process) but that is trivial to do.

On reflection I don't think there is an issue with the garbage
collector if you adopted this approach.  After forking there is only
one thread running in the new process - the thread of execution of the
forking thread - and provided that the new process does not attempt to
allocate memory after the fork and before the exec, I doubt the garbage
collector has a way in which it can be provoked to begin trying to
reclaim memory in the new process.