Re: request for review: Doing direct file I/O in Emacs Lisp

[David Kastrup <dak@gnu.org>]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: David Kastrup <dak@gnu.org>
> > Date: 16 May 2004 00:13:57 +0200
> > 
> > The problem is that I/O using "select" is ready the moment a _single_
> > byte is available.
> 
> Forgive me for a possibly stupid question, but with applications such
> as `cat' with their stdout redirected to a pipe, shouldn't we have
> `cat' write the pipe in relatively large blocks, as defined by the
> default buffering of its stdout?

cat will probably write in lines when writing to a pseudo tty.

Anyway, it is not as much cat I am concerned with mostly: the most
common problem are applications with small buffer size running in
comint mode.

Running TeX inside of an Emacs buffer is slow, easily 30% slower than
running it in an XTerm.  And it is the console I/O that causes this.

> > Perhaps one would need some nicer system calls for telling Linux
> > "ok, wake me up immediately if any pipe is _full_.
> 
> Almost full, you mean.  If the pipe is full, it's too late, since
> the pipe could be written again before the pipe reader actually
> awakes and runs, in which case you will lose characters at best and
> get SIGPIPE at worst.

I recommend that you look up the semantics of Unix pipes again.  You
seem to labor under the delusion that there is a constant race
condition going on.  If the pipe is filled, the writing process will
simply get blocked.  SIGPIPE is never raised unless the reading
process has _closed_ its end of the pipe, for example by exiting.

> Also, this has complications when the pipe writer exits without
> filling the pipe (which is the normal case).

Again, I don't know what kind of pipes you are talking of here.  A
read call on a pipe will block the reader until enough data is
available, or until the writer has closed its end of the pipe.

> > And wake me up immediately if there is input on some of [list of
> > files].  Other than that, only wake me up if there is input and no
> > other process is wanting the CPU".  So we'd need to tell the
> > operating system how urgent we want what amount of data on what
> > input to be scheduled for processing.
> 
> Sounds like a brutal intervention into the OS scheduler, which most
> systems will not allow.

Intervention?  Telling the scheduler where I expect interactive
response, and where I expect batch processing?  That is no
"intervention", it is information.

> It's possible that a much simpler solution would be to have a
> separate thread or a helper subprocess read the pipe until a certain
> amount of data is available or a timeout expires, and only then
> present the data to Emacs's process filter/sentinel.

Maybe.  If we had some fast preprocessor that would score only a
moderate penalty when the operating system schedules it all the time,
this might help quite a bit.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum