Re: request for review: Doing direct file I/O in Emacs Lisp

[David Kastrup <dak@gnu.org>]

John Wiegley <johnw@gnu.org> writes:

> If I start a process with start-process (/usr/bin/cat) and redirect
> its output to a file, it is far slower than if I simply output the
> same data to a file (eshell/cat) -- even though the resulting
> "output" in both cases is the same.  Why is receiving output via a
> process sentinel so slow?

Use a multiprocessor machine.

Part of the reason is the same because of which
process-adaptive-read-buffering exists:

cat produces some output.  As soon as one line (or whatever unit) is
full, the operating system intervenes and schedules Emacs.  Emacs
processes one line of output, then checks whether it can run timers,
whatever.  If Emacs finally has decided it can't do anything more
useful, it puts itself to sleep, reading on the pipe.  Then the
operating system wakes up cat again, for another single line.

The main fault, in my opinion, lies with the "low-latency" operating
system that schedules away the CPU from the writing process as soon as
it has produced any output.  That makes pipes pretty inefficient.  If
you have a multiprocessor machine, a simple job like "cat" can easily
stuff the pipe completely while Emacs is processing the last chunk.
On a uniprocessor machine, this does not happen since cat does not
even get the tiny amount of CPU power necessary to fill the pipe.

The problem is that I/O using "select" is ready the moment a _single_
byte is available.

Perhaps one would need some nicer system calls for telling Linux "ok,
wake me up immediately if any pipe is _full_.  And wake me up
immediately if there is input on some of [list of files].  Other than
that, only wake me up if there is input and no other process is
wanting the CPU".  So we'd need to tell the operating system how
urgent we want what amount of data on what input to be scheduled for
processing.

process-adaptive-read-buffering tries to fudge around this problem.  I
have some feeling that it might still be buggy.  I seem to remember
some inconclusive reports where larger delays occured, maybe under
MSWindows.

Basically, reports indicate that even with
process-adaptive-read-buffering one is maybe 30% slower than if the
command gets started with a trailing "|dd obs=8k" pipeline, but still
quite faster than if you don't use  process-adaptive-read-buffering.

A mess.

Anyway, it might be worth optimizing and profiling Emacs for good
typical filter routine performance even when small data chunks are
involved.  The more administrational overhead Emacs tries to do on
each little wakeup, the slower we get.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum