Proposal for extending set-process-filter

Proposal for extending set-process-filter

[David Kastrup]

Currently, processes called with call-process can't have their stdout
and stderr directed to different buffers/filter functions/whatever.
This is painfully obvious when using pipes and redirections in eshell.

I'd propose that set-process-filter gets extended in the following
way:

set-process-filter is a built-in function in `src/process.c'.
(set-process-filter PROCESS FILTER)

Give PROCESS the filter function FILTER; nil means no filter.
t means stop accepting output from the process.

A filter can be 

1) a function which gets two arguments. [insert old description]
2) a buffer into which output gets inserted.
3) a file name for output.
4) t to stop accepting output.
5) a list of filter/file descriptor associations.
The car's of the elements of this list are an item from 1)-4) as
above, while the cdr is a list of file ids for which this item is
supposed to apply.

So to redirect output of stdout and stderr to /dev/null, you'd specify
(set-process-filter my-process '(("/dev/null" 1 2)))

If a list is specified in this manner, the operation of unspecified
file descriptors is not affected.

This does not yet do everything needed for a full-bodied eshell: input
redirections (including procedural input, equivalents to filter
functions) and output appending redirection (>>) are not covered
yet.  But it would be a first step.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Proposal for extending set-process-filter

[Stefan Monnier]

> Give PROCESS the filter function FILTER; nil means no filter.

What happens to output when there's no filter?

> 1) a function which gets two arguments. [insert old description]

Good.

> 2) a buffer into which output gets inserted.

Good.

> 3) a file name for output.

Can be handled by (1) just fine.  What would you use it for (other than
/dev/null which is rather special since append is the same as overwrite)?

> 4) t to stop accepting output.

What happens to the output?  Is it read&thrown away or not read?

> 5) a list of filter/file descriptor associations.
> The car's of the elements of this list are an item from 1)-4) as
> above, while the cdr is a list of file ids for which this item is
> supposed to apply.

Sounds good.

> This does not yet do everything needed for a full-bodied eshell: input
> redirections (including procedural input, equivalents to filter
> functions) and output appending redirection (>>) are not covered
> yet.  But it would be a first step.

Output-appending can be done via a filter-function.
What kind of procedural input are you thinking of that can't be done via
process-send-string?

What is still missing is the ability to open more file descriptors than
stdin, stdout, stderr.


        Stefan

Re: Proposal for extending set-process-filter

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Give PROCESS the filter function FILTER; nil means no filter.
> 
> What happens to output when there's no filter?

It is inserted into the process buffer at point (default behavior).

> > 1) a function which gets two arguments. [insert old description]
> 
> Good.
> 
> > 2) a buffer into which output gets inserted.
> 
> Good.
> 
> > 3) a file name for output.
> 
> Can be handled by (1) just fine.  What would you use it for (other than
> /dev/null which is rather special since append is the same as overwrite)?

Consistency with the BUFFER argument of start-process (which would be
pretty much the same as the FILTER function, with the exception that
it establishes a control buffer that is the default for process
output, that will list the process as one of its processes, and that
will kill the process if the buffer gets killed).  Also efficiency:
redirecting a file descriptor to a file does not require Emacs
processing power.  It also does not require conversion into multibyte
strings and back and so on.

> > 4) t to stop accepting output.
> 
> What happens to the output?  Is it read&thrown away or not read?

Neither.  File descriptor is closed.  That means that there is no
output to accept: write calls of the generating process will fail
with something like EPIPE.

> > 5) a list of filter/file descriptor associations.  The car's of
> > the elements of this list are an item from 1)-4) as above, while
> > the cdr is a list of file ids for which this item is supposed to
> > apply.
> 
> Sounds good.
> 
> > This does not yet do everything needed for a full-bodied eshell: input
> > redirections (including procedural input, equivalents to filter
> > functions) and output appending redirection (>>) are not covered
> > yet.  But it would be a first step.
> 
> Output-appending can be done via a filter-function.

Not really.  An open in append mode will also append at the end if
the file gets extended by a different process.  That's pretty hard to
do with a filter function.  One could start the file descriptor list
with
:append t
or so, however.

> What kind of procedural input are you thinking of that can't be done
> via process-send-string?

process-send-string blocks when a pipe is full.  Also it only works
on file descriptor 0, ever.  If I do something like a shell
redirection:

some command <<EOF &
lots of text
EOF

then eshell would block if "lots of text" would be more than a pipe's
worth.

> What is still missing is the ability to open more file descriptors
> than stdin, stdout, stderr.

Item 5) above was supposed to cater for that.  What really is missing
is the ability to open a pipe that will, again, work without Emacs'
intervention.  Not sure how to do this.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Proposal for extending set-process-filter

[Stefan Monnier]

>> > 3) a file name for output.
>> Can be handled by (1) just fine.  What would you use it for (other than
>> /dev/null which is rather special since append is the same as overwrite)?
> Consistency with the BUFFER argument of start-process (which would be
> pretty much the same as the FILTER function, with the exception that
> it establishes a control buffer that is the default for process
> output, that will list the process as one of its processes, and that
> will kill the process if the buffer gets killed).  Also efficiency:
> redirecting a file descriptor to a file does not require Emacs
> processing power.  It also does not require conversion into multibyte
> strings and back and so on.

AFAIK, you can't do such a redirection after the exec.  I.e. it has to be an
argument of `start-process' rather than set-process-filter.

>> Output-appending can be done via a filter-function.
> Not really.  An open in append mode will also append at the end if
> the file gets extended by a different process.  That's pretty hard to
> do with a filter function.  One could start the file descriptor list
> with
> :append t
> or so, however.

I thought the `append' arg of write-region did it right.

>> What is still missing is the ability to open more file descriptors
>> than stdin, stdout, stderr.
> Item 5) above was supposed to cater for that.

But for the same reason as above I don't think this can be done in
set-process-filter: it has to be done before the `exec'.


        Stefan

Re: Proposal for extending set-process-filter

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >> > 3) a file name for output.
> >> Can be handled by (1) just fine.  What would you use it for (other than
> >> /dev/null which is rather special since append is the same as overwrite)?
> > Consistency with the BUFFER argument of start-process (which would be
> > pretty much the same as the FILTER function, with the exception that
> > it establishes a control buffer that is the default for process
> > output, that will list the process as one of its processes, and that
> > will kill the process if the buffer gets killed).  Also efficiency:
> > redirecting a file descriptor to a file does not require Emacs
> > processing power.  It also does not require conversion into multibyte
> > strings and back and so on.
> 
> AFAIK, you can't do such a redirection after the exec.  I.e. it has
> to be an argument of `start-process' rather than set-process-filter.

Ouch.  I am so stupid.  In that case, the whole folderol obviously
needs to be made some option right at start-process time.

I'll have to think more about this.

> > Not really.  An open in append mode will also append at the end if
> > the file gets extended by a different process.  That's pretty hard to
> > do with a filter function.  One could start the file descriptor list
> > with
> > :append t
> > or so, however.
> 
> I thought the `append' arg of write-region did it right.

Probably.  But it would be quite wasteful to use Emacs for that.  But
since I forgot in my stupidity that once the fork has happened, we
can't work on anything but the pipe, of course one can't avoid Emacs
reading on the pipe.

Again, the _intent_ of the proposal was to allow redirections and
stuff like in a shell, and also different filter functions on
different file descriptors.

Naturally, it fails.  The stuff needs to be specified before exec.

> But for the same reason as above I don't think this can be done in
> set-process-filter: it has to be done before the `exec'.

Yes.

I am glad that there is somebody with a brain on one end of our
connection.

So obviously, this would need to be changed into a proposal for
start-process.  The BUFFER argument might be replaced by a cons cell
where the CAR is the previous BUFFER argument (meaning the buffer that
the process will get associated with), and the CDR is the contraption
that I proposed for set-process-filter.  set-process-filter could
still work as described, but it obviously would only be allowed to
apply to file descriptors under Emacs control (namely, pointing to a
buffer or filter function), and not file descriptors closed or
redirected to a file.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Proposal for extending set-process-filter

[Richard Stallman]

    1) a function which gets two arguments. [insert old description]
    2) a buffer into which output gets inserted.

You should use set-process-buffer for that.

    3) a file name for output.

Writing it directly into a file is somewhat ugly as an idea.

    4) t to stop accepting output.

Why do we need that option?  A filter function can easily discard the
data.

    5) a list of filter/file descriptor associations.
    The car's of the elements of this list are an item from 1)-4) as
    above, while the cdr is a list of file ids for which this item is
    supposed to apply.

That is very ugly.  There has to be a better way.

Anyway, the hard part of separating stdin from stderr
is that they both come through the same channel in the pty.

Re: Proposal for extending set-process-filter

[Stefan Monnier]

>     1) a function which gets two arguments. [insert old description]
>     2) a buffer into which output gets inserted.
> You should use set-process-buffer for that.

I think he only included this case for use in his (5) where it does make
sense if you want one descriptor to go to buffer A and another to buffer B.


        Stefan

Re: Proposal for extending set-process-filter

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >     1) a function which gets two arguments. [insert old description]
> >     2) a buffer into which output gets inserted.
> > You should use set-process-buffer for that.

No, set-process-buffer actually changes the _associated_ buffer.
That is the buffer that when killed, kills the process as well.  It
is the buffer in which get-buffer-process might return the buffer.
It incidentally is also the _default_ location where output arrives
in case we have no filter function.

What I proposed was intended to redirect output (possibly just on one
file descriptor of stdout and stderr) to a potentially _different_
buffer.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Proposal for extending set-process-filter

[Richard Stallman]

    What I proposed was intended to redirect output (possibly just on one
    file descriptor of stdout and stderr) to a potentially _different_
    buffer.

In general, when the output from a process goes into a buffer, we want
the process to have that buffer as its process buffer.  In the
exceptional case where we do not want that buffer to be its process
buffer, we could easily write a filter function to insert the output
into a buffer.

Re: Proposal for extending set-process-filter

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     1) a function which gets two arguments. [insert old description]
>     2) a buffer into which output gets inserted.
> 
> You should use set-process-buffer for that.

That is used for the _controlling_ buffer associated with the process
(which incidentally is also the default where to insert stuff when no
other output is given).

>     3) a file name for output.
> 
> Writing it directly into a file is somewhat ugly as an idea.

Definitely.  I was confused into thinking that one could redirect the
file descriptors directly, but that can't happen after
start-process.  I did a revamped proposal to take care of that.  See
separate post.

>     4) t to stop accepting output.
> 
> Why do we need that option?  A filter function can easily discard the
> data.

Closing a file and discard output are two different things.  For
example, the "yes" command line utility will stop once its output is
closed, but will continue to infinitely produce output if it is only
thrown away.

Basically, I am again mostly thinking of eshell here: again, the
utility is minor after start-process has already been done.

>     5) a list of filter/file descriptor associations.  The car's of
>     the elements of this list are an item from 1)-4) as above, while
>     the cdr is a list of file ids for which this item is supposed to
>     apply.
> 
> That is very ugly.  There has to be a better way.
> 
> Anyway, the hard part of separating stdin from stderr
> is that they both come through the same channel in the pty.

set-process-filter is too late for this.  But if we do that at
start-process-time, we don't need to use a pty here except maybe for
stdout when specified in the ordinary manner?

Commands that don't specify different buffers/filter
functions/whatever for stdout and stderr can still receive them
through a single channel.

For that idea to make sense, the separation of course would have to
happen at start-process time already.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Proposal for extending set-process-filter

[Stefan Monnier]

>> Why do we need that option?  A filter function can easily discard the
>> data.

> Basically, I am again mostly thinking of eshell here: again, the
> utility is minor after start-process has already been done.

It think it makes sense to provide a way to close a process's
file descriptor.  We already have process-send-eof to close stdin.


        Stefan