Re: Any idea about what makes Emacs slow reading on pipes?

[David.Kastrup@t-online.de (David Kastrup)]

storm@cua.dk (Kim F. Storm) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:

> So I do see the "1 byte" reads, but not the slowness.

Fine for you.

> > I don't know _what_ Emacs does do the pipe/pty that causes only single
> > characters to gather/be processed, with a sporadic 1024 packet getting
> > through from time to time, but I hate it.  Some setting must be on the
> > pipe that causes the writer to stall before further writes in many
> > cases until Emacs has read the next character.  And after some time
> > the pipe gets filled properly now and then, with single characters in
> > between.
> > 
> > I don't get it.
> 
> Well, it seems that emacs is FAST enough the keep up with your dd
> which actually only writes 1 byte at a time, so basically if your
> program writes 1 char at a time, emacs will read 1 char at a time if
> it can keep up...

No, that isn't it.  I can pipe the stuff into |cat >/dev/null instead
of Emacs, and it will be finished in a whiffy.  Emacs must be using
some sort of system call that _stops_ dd most of the time from being
able to so another write to the pipe until the Linux scheduler has run
again (which it does at the rate of about 100 per second, almost
exactly the rate at which characters arrive).  I am not saying that
the current Linux kernel might not play its part in the catastrophic
performance here, but the fact is that substituting different readers
for Emacs on the delivering pipe yields in vast performance gains of
the _writing_ process.

> What happens if you try the following version, which forces
> start-process to use a pipe rather than a pty:

Nothing much, I said already.

> But if you really want to speed up things, use a block size
> of 1k like this:

> 	       "-c" "hexdump -v /dev/zero|dd bs=1k count=100")

Oh, come on.  The dd was explicitly there to force small writes,
since that is the most common situation when talking with
applications through a pipe.  And that is what makes Emacs dog slow
when compared with XEmacs when running things like TeX from within
AUCTeX.

> Based on this, I would conclude that emacs is behaving _correctly_
> for the input you supply it.

Well, it certainly does receive and process all bytes, so of _course_
it is behaving `correctly'.  But it causes dd and perhaps Linux to
crawl.  I am not interested in assigning a culprit, but the fact
remains that the implementation causes awful performance.  Part of the
reason is that Emacs does a lot of partly excessively looking overhead
for every accept-process-output or related call (like sit-for).  But
improving that would not pay off even remotely as well as having a way
to let the pipe have better utilization.  If Emacs could find a way
_not_ to let Linux wake it up until the writing process either is
regularly preempted, or has the pipe filled, or is waiting for
something other, this would greatly benefit performance by causing
Emacs to work on more useful batches than just single characters.

We are having here an operating system with a single CPU, and we
switch processes here for every byte mainly, and the producing process
gets no CPU time for making new bytes while the consuming process is
still busy processing the byte it got.  Anyhow, I _do_ have the
additional suspicion that there is pretty much one "tick" of delay for
every of byte involved and that the CPU is mostly idling for that
reason: a 600MHz PIII should even with the full processing overhead
due to catering for every single character on its own be able to
handle more than 100 characters per second.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum