Re: Any idea about what makes Emacs slow reading on pipes?

[storm@cua.dk (Kim F. Storm)]

David.Kastrup@t-online.de (David Kastrup) writes:

> No, that isn't it.  I can pipe the stuff into |cat >/dev/null instead
> of Emacs, and it will be finished in a whiffy.  Emacs must be using
> some sort of system call that _stops_ dd most of the time from being
> able to so another write to the pipe until the Linux scheduler has run
> again (which it does at the rate of about 100 per second, almost
> exactly the rate at which characters arrive).  

That is because emacs is using a pty by default for process i/o (I
guess that is to avoid buffering output from the writer, so emacs will
get a more steady flow of input to process, e.g. for compiler output),
and it sets the pty to non-blocking mode when reading from the pty 
(to avoid having emacs hanging on the read system call until it has
successfully read all 1024 bytes).

The combined effect is that the write (such as tex) will use
un-buffered writes, and emacs will use non-blocing reads, effectively
using as small data units between writer and reader as possible.

If you just do your-command | cat > /dev/null, your-command will
most likely do buffered i/o, so it will surely run faster.

>                                                I am not saying that
> the current Linux kernel might not play its part in the catastrophic
> performance here, but the fact is that substituting different readers
> for Emacs on the delivering pipe yields in vast performance gains of
> the _writing_ process.
> 
> > What happens if you try the following version, which forces
> > start-process to use a pipe rather than a pty:
> 
> Nothing much, I said already.

Well, it could have an effect for your ACUTeX case, since it would
then be writing to a pipe (using buffered i/o) rather than to a pty
(typically using non-buffered i/o).

> > But if you really want to speed up things, use a block size
> > of 1k like this:
> 
> > 	       "-c" "hexdump -v /dev/zero|dd bs=1k count=100")
> 
> Oh, come on.  

Come on yourself...  This illustrates that if you use
        "tex ... | dd bs=1k"
instead of just
        "tex ..."
in acutex, you'll probably end up with much faster performance
since you will get buffered output from tex.

But you should get the same effect by setting process-connection-type
to nil.


>               The dd was explicitly there to force small writes,
> since that is the most common situation when talking with
> applications through a pipe.  

I would expect that for a pty, not for a pipe!

>                               And that is what makes Emacs dog slow
> when compared with XEmacs when running things like TeX from within
> AUCTeX.

Maybe Xemacs is using pipes rather than ptys by default (or never use
pty's at all)...

> 
> We are having here an operating system with a single CPU, and we
> switch processes here for every byte mainly, and the producing process
> gets no CPU time for making new bytes while the consuming process is
> still busy processing the byte it got.  Anyhow, I _do_ have the
> additional suspicion that there is pretty much one "tick" of delay for
> every of byte involved and that the CPU is mostly idling for that
> reason: a 600MHz PIII should even with the full processing overhead
> due to catering for every single character on its own be able to
> handle more than 100 characters per second.

Could you try

        M-: (setq process-connection-type nil)

before you run your acutex process, and see if that makes a difference.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk