Any idea about what makes Emacs slow reading on pipes?

Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

The following exhibits abysmal speed within Emacs:

(let (process-connection-type) (switch-to-buffer (generate-new-buffer
"*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
"-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))

More to the point, things start out dead slow and get faster later.
It is not the fault of the programs on the sending side: just piping
into cat >/dev/null instead of Emacs is much much faster.  It does
not seem that setting process-connection-type to nil as above (using
a pipe instead of a pty) does help worth noting.

The system I see this in is
GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
a RedHat 9 system, but I have seen this in practically every Linux
system up to now.  Is this a Linux-specific problem, or do others see
this as well?

I already reported this once as a bug, but have seen no reply
whatsoever.  This is _really_ impacting using Emacs as a shell for
anything, as well as process I/O in the background (like that of
preview-latex which runs appallingly slower on Emacs than on XEmacs
due to this bug).  Anybody that knows about the process system that
would have a clue about what might be going on here?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Andreas Schwab]

David.Kastrup@t-online.de (David Kastrup) writes:

|> The following exhibits abysmal speed within Emacs:
|> 
|> (let (process-connection-type) (switch-to-buffer (generate-new-buffer
|> "*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
|> "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))
|> 
|> More to the point, things start out dead slow and get faster later.
|> It is not the fault of the programs on the sending side: just piping
|> into cat >/dev/null instead of Emacs is much much faster.  It does
|> not seem that setting process-connection-type to nil as above (using
|> a pipe instead of a pty) does help worth noting.
|> 
|> The system I see this in is
|> GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
|> a RedHat 9 system, but I have seen this in practically every Linux
|> system up to now.  Is this a Linux-specific problem, or do others see
|> this as well?

I cannot reproduce that on ppc-suse-linux.  I also tried out MacOS X on
the same machine, and there the overall speed is much lower, but still
mostly constant.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Any idea about what makes Emacs slow reading on pipes?

[Kevin Rodgers]

David Kastrup wrote:

> The following exhibits abysmal speed within Emacs:
> 
> (let (process-connection-type) (switch-to-buffer (generate-new-buffer
> "*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
> "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))
> 
> More to the point, things start out dead slow and get faster later.
> It is not the fault of the programs on the sending side: just piping
> into cat >/dev/null instead of Emacs is much much faster.  It does
> not seem that setting process-connection-type to nil as above (using
> a pipe instead of a pty) does help worth noting.
> 
> The system I see this in is
> GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
> a RedHat 9 system, but I have seen this in practically every Linux
> system up to now.  Is this a Linux-specific problem, or do others see
> this as well?

/usr/bin/sh: hexdump: not found

Can you suggest an alternative to hexdump for Solaris?

-- 
<a href="mailto:&lt;kevin.rodgers&#64;ihs.com&gt;">Kevin Rodgers</a>

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

Kevin Rodgers <ihs_4664@yahoo.com> writes:

> David Kastrup wrote:
> 
> > The following exhibits abysmal speed within Emacs:
> > (let (process-connection-type) (switch-to-buffer
> > (generate-new-buffer
> > "*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
> > "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))
> > More to the point, things start out dead slow and get faster later.
> > It is not the fault of the programs on the sending side: just piping
> > into cat >/dev/null instead of Emacs is much much faster.  It does
> > not seem that setting process-connection-type to nil as above (using
> > a pipe instead of a pty) does help worth noting.
> > The system I see this in is
> > GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
> > a RedHat 9 system, but I have seen this in practically every Linux
> > system up to now.  Is this a Linux-specific problem, or do others see
> > this as well?
> 
> /usr/bin/sh: hexdump: not found
> 
> Can you suggest an alternative to hexdump for Solaris?

Probably od -h -v or something.  The point was just fast producing
large amounts of more or less reproducable text.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Jan D.]

>
> The following exhibits abysmal speed within Emacs:
>
> (let (process-connection-type) (switch-to-buffer (generate-new-buffer
> "*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
> "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))

...

> The system I see this in is
> GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
> a RedHat 9 system, but I have seen this in practically every Linux
> system up to now.  Is this a Linux-specific problem, or do others see
> this as well?

I could not reproduce this on a slackware (version unknown, but libc 6)
or a Mandrake 9.0 system.  Maybe running under strace can give some
indication of the system call that takes time (if that is the issue)?

	Jan D.

Re: Any idea about what makes Emacs slow reading on pipes?

[Kevin Rodgers]

David Kastrup wrote:

> Probably od -h -v or something.  The point was just fast producing
> large amounts of more or less reproducable text.


Using "od -v" runs fast from the start on GNU Emacs 21.3.1 (i386-pc-solaris2.7, X toolkit).

-- 
<a href="mailto:&lt;kevin.rodgers&#64;ihs.com&gt;">Kevin Rodgers</a>

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

"Jan D." <jan.h.d@swipnet.se> writes:

> >
> > The following exhibits abysmal speed within Emacs:
> >
> > (let (process-connection-type) (switch-to-buffer (generate-new-buffer
> > "*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
> > "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))
> 
> ...
> 
> > The system I see this in is
> > GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
> > a RedHat 9 system, but I have seen this in practically every Linux
> > system up to now.  Is this a Linux-specific problem, or do others see
> > this as well?
> 
> I could not reproduce this on a slackware (version unknown, but libc 6)
> or a Mandrake 9.0 system.  Maybe running under strace can give some
> indication of the system call that takes time (if that is the issue)?

Here is a more elaborate test file

[-- Attachment #2: Check filter function strangeness --]
[-- Type: application/emacs-lisp, Size: 552 bytes --]

[-- Attachment #3: Type: text/plain, Size: 3000 bytes --]


If I run this and print out test-pattern afterwards, I get

test-pattern

((485 0 11) (1024 0 11) (1024 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0
11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1
0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11)
(1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1023 0 11) (1024 0 11)
(1024 0 11) (1024 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11) (1 0 11)
(1 0 11) (1 0 11) (1 0 11) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0
10) (1 0 10) (1023 0 10) (1024 0 10) (1024 0 10) (1024 0 10) (1 0 10)
(1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0
10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1
0 10) (1 0 10) (1 0 10) (1023 0 10) (1024 0 10) (1024 0 10) (1024 0
10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1
0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10)
(1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1 0 10) (1023 0 10)

[...]

(1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0
8) (1 0 8) (1 0 8) (1 0 8) (1023 0 8) (1024 0 8) (1024 0 8) (1024 0 8)
(1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0
8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 8) (1 0 7) (1
0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7)
(1023 0 7) (1024 0 7) (1024 0 7) (1024 0 7) (1 0 7) (1 0 7) (1 0 7) (1
0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7)
(1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0
7) (1 0 7) (1 0 7) (1 0 7) (1023 0 7) (1024 0 7) (1024 0 7) (1024 0 7)
(1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0
7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1 0 7) (1

[...]

0 6) (1 0 6) (1 0 6) (1 0 6) (1023 0 6) (1024 0 6) (1024 0 6) (1024 0
6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1
0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6)
(1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0
6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1
0 6) (1023 0 6) (1024 0 6) (1024 0 6) (1024 0 6) (1 0 6) (1 0 6) (1 0
6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1 0 6) (1

And from then on (actually backwards in time) only packets with single
characters, with perhaps 100 filter calls per second.

Which in itself is not too hot a speed, but gets awful in consequence
because the filter routine gets only called for single characters.

I don't know _what_ Emacs does do the pipe/pty that causes only single
characters to gather/be processed, with a sporadic 1024 packet getting
through from time to time, but I hate it.  Some setting must be on the
pipe that causes the writer to stall before further writes in many
cases until Emacs has read the next character.  And after some time
the pipe gets filled properly now and then, with single characters in
between.

I don't get it.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

Kevin Rodgers <ihs_4664@yahoo.com> writes:

> David Kastrup wrote:
> 
> > Probably od -h -v or something.  The point was just fast producing
> > large amounts of more or less reproducable text.
> 
> 
> Using "od -v" runs fast from the start on GNU Emacs 21.3.1
> (i386-pc-solaris2.7, X toolkit).

What's the distribution of filter lengths with the separate test
program I posted?  Something less extreme than switching between 1 and
1024?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Jan D.]

> Here is a more elaborate test file
> <testio.el>
> If I run this and print out test-pattern afterwards, I get
>
> test-pattern
...

>
> And from then on (actually backwards in time) only packets with single
> characters, with perhaps 100 filter calls per second.

I think this is a scheduling issue.  After all, dd is only reading one
character at a time.  I do get some one character reads, but not many.
There is more one character reads on a RedHat 6 than on the Mandrake 9,
not because of some difference in the systems, but because the RedHat
runs on a much slower machine (AMD K6 450Mhz versus AMD Athlon 1400).

For example, if I change to dd bs=1024 count=100, I get no single 
character
reads at all.  If I raise the priority of dd, I get 1024 consistently 
even
on the slower machine.

	Jan D.

Re: Any idea about what makes Emacs slow reading on pipes?

[Andreas Schwab]

David.Kastrup@t-online.de (David Kastrup) writes:

|> "Jan D." <jan.h.d@swipnet.se> writes:
|> 
|> > >
|> > > The following exhibits abysmal speed within Emacs:
|> > >
|> > > (let (process-connection-type) (switch-to-buffer (generate-new-buffer
|> > > "*test*"))(erase-buffer)(start-process "test" (current-buffer) "sh"
|> > > "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")(erase-buffer))
|> > 
|> > ...
|> > 
|> > > The system I see this in is
|> > > GNU Emacs 21.3.50.3 (i686-pc-linux-gnu)
|> > > a RedHat 9 system, but I have seen this in practically every Linux
|> > > system up to now.  Is this a Linux-specific problem, or do others see
|> > > this as well?
|> > 
|> > I could not reproduce this on a slackware (version unknown, but libc 6)
|> > or a Mandrake 9.0 system.  Maybe running under strace can give some
|> > indication of the system call that takes time (if that is the issue)?
|> 
|> Here is a more elaborate test file

Tried on ppc-suse-linux, I get constant throughput of 1024 characters at a
time.  On MacOS X (through MOL) I get varying packet sizes between 97 and
1024.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Any idea about what makes Emacs slow reading on pipes?

[Satyaki Das]

David Kastrup <David.Kastrup@t-online.de> writes:

> Here is a more elaborate test file
> 
> If I run this and print out test-pattern afterwards, I get
> 
> test-pattern

I ran the test and the result was very similar to yours. My system
is a Red Hat GNU/Linux 9.0 running on a 800 MHz Pentium 3. The CVS
version of Emacs is from yesterday.

Satyaki

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

storm@cua.dk (Kim F. Storm) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:

> So I do see the "1 byte" reads, but not the slowness.

Fine for you.

> > I don't know _what_ Emacs does do the pipe/pty that causes only single
> > characters to gather/be processed, with a sporadic 1024 packet getting
> > through from time to time, but I hate it.  Some setting must be on the
> > pipe that causes the writer to stall before further writes in many
> > cases until Emacs has read the next character.  And after some time
> > the pipe gets filled properly now and then, with single characters in
> > between.
> > 
> > I don't get it.
> 
> Well, it seems that emacs is FAST enough the keep up with your dd
> which actually only writes 1 byte at a time, so basically if your
> program writes 1 char at a time, emacs will read 1 char at a time if
> it can keep up...

No, that isn't it.  I can pipe the stuff into |cat >/dev/null instead
of Emacs, and it will be finished in a whiffy.  Emacs must be using
some sort of system call that _stops_ dd most of the time from being
able to so another write to the pipe until the Linux scheduler has run
again (which it does at the rate of about 100 per second, almost
exactly the rate at which characters arrive).  I am not saying that
the current Linux kernel might not play its part in the catastrophic
performance here, but the fact is that substituting different readers
for Emacs on the delivering pipe yields in vast performance gains of
the _writing_ process.

> What happens if you try the following version, which forces
> start-process to use a pipe rather than a pty:

Nothing much, I said already.

> But if you really want to speed up things, use a block size
> of 1k like this:

> 	       "-c" "hexdump -v /dev/zero|dd bs=1k count=100")

Oh, come on.  The dd was explicitly there to force small writes,
since that is the most common situation when talking with
applications through a pipe.  And that is what makes Emacs dog slow
when compared with XEmacs when running things like TeX from within
AUCTeX.

> Based on this, I would conclude that emacs is behaving _correctly_
> for the input you supply it.

Well, it certainly does receive and process all bytes, so of _course_
it is behaving `correctly'.  But it causes dd and perhaps Linux to
crawl.  I am not interested in assigning a culprit, but the fact
remains that the implementation causes awful performance.  Part of the
reason is that Emacs does a lot of partly excessively looking overhead
for every accept-process-output or related call (like sit-for).  But
improving that would not pay off even remotely as well as having a way
to let the pipe have better utilization.  If Emacs could find a way
_not_ to let Linux wake it up until the writing process either is
regularly preempted, or has the pipe filled, or is waiting for
something other, this would greatly benefit performance by causing
Emacs to work on more useful batches than just single characters.

We are having here an operating system with a single CPU, and we
switch processes here for every byte mainly, and the producing process
gets no CPU time for making new bytes while the consuming process is
still busy processing the byte it got.  Anyhow, I _do_ have the
additional suspicion that there is pretty much one "tick" of delay for
every of byte involved and that the CPU is mostly idling for that
reason: a 600MHz PIII should even with the full processing overhead
due to catering for every single character on its own be able to
handle more than 100 characters per second.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Kim F. Storm]

David.Kastrup@t-online.de (David Kastrup) writes:

> Here is a more elaborate test file
> 
> (defvar test-pattern nil)
> (defvar test-start nil)
> 
> (defun test-filter (process string)
>   (push (cons (length string) (time-since test-start)) test-pattern)
>   (with-current-buffer (process-buffer process)
>     (insert-before-markers string)))
> 
> (defun make-test nil (interactive)
>   (switch-to-buffer (get-buffer-create "*test*"))
>   (erase-buffer)
>   (setq test-pattern nil test-start (current-time))
>   (set-process-filter (start-process
> 		       "test" (current-buffer) "sh"
> 		       "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")
> 		      #'test-filter))
> 

With your test program, I get the following result on redhat 6.2 on a
650HMz P3 system:

((87 0 2) (1024 0 2) (1024 0 2) (1023 0 2) (1024 0 2) (1024 0 2) (1024
0 2) (1023 0 2) (1024 0 2) (1024 0 2) (1024 0 2) (1023 0 2) (1024 0 2)
(1024 0 2) (1024 0 2) (1023 0 2) (1024 0 2) (1024 0 2) (1024 0 2)
(1023 0 2) (1024 0 2) (1024 0 2) (1024 0 2) (1023 0 2) (1024 0 2)
(1024 0 2) (1024 0 2) (1023 0 2) (1024 0 2) (1024 0 2) (1024 0 2)
(1023 0 2) (1024 0 2) (1024 0 2) (1024 0 1) (1023 0 1) (1024 0 1)
(1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1)
(1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1)
(1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1)
(1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1)
(1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1)
(1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1)
(1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1)
(1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1)
(1024 0 1) (1024 0 1) (1023 0 1) (1024 0 1) (1024 0 1) (1024 0 1)
(1024 0 1) (1024 0 1) (1024 0 1) (1024 0 1) (1024 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (304 0 0) (1024 0 0) (1 0 0)
(1 0 0) (582 0 0) (1024 0 0) (1024 0 0) (1024 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0)
(1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0) (1 0 0))

So I do see the "1 byte" reads, but not the slowness.

> I don't know _what_ Emacs does do the pipe/pty that causes only single
> characters to gather/be processed, with a sporadic 1024 packet getting
> through from time to time, but I hate it.  Some setting must be on the
> pipe that causes the writer to stall before further writes in many
> cases until Emacs has read the next character.  And after some time
> the pipe gets filled properly now and then, with single characters in
> between.
> 
> I don't get it.

Well, it seems that emacs is FAST enough the keep up with your dd
which actually only writes 1 byte at a time, so basically if your
program writes 1 char at a time, emacs will read 1 char at a time if
it can keep up...  I guess the occasional 1024 read is because of
scheduling of other processes or some such, and because eventually,
emacs cannot keep up anymore (when it has to extend the buffer or some
such).

What happens if you try the following version, which forces
start-process to use a pipe rather than a pty:

(defun make-test nil (interactive)
  (switch-to-buffer (get-buffer-create "*test*"))
  (erase-buffer)
  (setq test-pattern nil test-start (current-time))
  (let ((process-connection-type nil))
    (set-process-filter (start-process
	       "test" (current-buffer) "sh"
	       "-c" "hexdump -v /dev/zero|dd bs=1 count=100k")
	      #'test-filter)))

It has some effect in my setup, but nothing dramatic.

But if you really want to speed up things, use a block size
of 1k like this:

(defun make-test nil (interactive)
  (switch-to-buffer (get-buffer-create "*test*"))
  (erase-buffer)
  (setq test-pattern nil test-start (current-time))
  (let ((process-connection-type nil))
    (set-process-filter (start-process
	       "test" (current-buffer) "sh"
	       "-c" "hexdump -v /dev/zero|dd bs=1k count=100")
	      #'test-filter)))

On my system, it gives the optimal behaviour:

((35 0 2) (1024 0 2) (1024 0 2) ... (1024 0 2))


Based on this, I would conclude that emacs is behaving _correctly_ for
the input you supply it.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Any idea about what makes Emacs slow reading on pipes?

[Stefan Monnier]

> > No, that isn't it.  I can pipe the stuff into |cat >/dev/null instead
> > of Emacs, and it will be finished in a whiffy.  Emacs must be using
> > some sort of system call that _stops_ dd most of the time from being
> > able to so another write to the pipe until the Linux scheduler has run
> > again (which it does at the rate of about 100 per second, almost
> > exactly the rate at which characters arrive).
> That is because emacs is using a pty by default for process i/o (I

But David pointed out several times that asking Emacs to use a pipe didn't
make any difference.

BTW, on my machine (266Mhz PII) the result of his test is basically always
1024.  If I put the process' output in a hidden buffer, I then get mostly
1024 but a few smaller entries, the smaller being 382 on the test run I did.
Looks like my Emacs is just not fast enough.

> Come on yourself...  This illustrates that if you use
>         "tex ... | dd bs=1k"
> instead of just
>         "tex ..."
> in acutex, you'll probably end up with much faster performance
> since you will get buffered output from tex.

But there's still something very fishy: he gets poor performance
while at the same time Emacs only reads one char at a time.
In your interpretation, the reason why only 1 char is read each time
is because Emacs is so fast that it keeps up with the input data, but
then why would the whole thing be so slow (considering that `dd' is
pretty fast when writing somewhere else, even with 1-byte-at-a-time
buffering).
So is it that Emacs ends up somehow slowing down `dd' to a crawl ?  How ?

>         M-: (setq process-connection-type nil)
> before you run your acutex process, and see if that makes a difference.

I think it's pretty clear that his test case is showing anomalous
behavior, based on his description.  And his test case doesn't
use AUCTeX.  I can't reproduce it, tho.


	Stefan

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

[-- Attachment #1: Type: text/plain, Size: 6632 bytes --]

storm@cua.dk (Kim F. Storm) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:
> 
> > No, that isn't it.  I can pipe the stuff into |cat >/dev/null
> > instead of Emacs, and it will be finished in a whiffy.  Emacs must
> > be using some sort of system call that _stops_ dd most of the time
> > from being able to so another write to the pipe until the Linux
> > scheduler has run again (which it does at the rate of about 100
> > per second, almost exactly the rate at which characters arrive).
> 
> That is because emacs is using a pty by default for process i/o (I
> guess that is to avoid buffering output from the writer, so emacs
> will get a more steady flow of input to process, e.g. for compiler
> output), and it sets the pty to non-blocking mode when reading from
> the pty (to avoid having emacs hanging on the read system call until
> it has successfully read all 1024 bytes).

Actually, this isn't it.  Pipes behave just the same, and I now know
why.  You won't see this problem on an SMP machine.  The writing
process makes a _small_ write, and Emacs, waiting on the receiving end
of it with select, gets woken up immediately and gets to process the
small write.  But while Emacs is processing the small write, the
writing process does not get any processing time at all, so it won't
produce anything new until Emacs either goes back to sleep, or is
preempted because of taking too long (for example garbage collecting).
The moment Emacs is preempted and not waiting on the select system
call, the writing process is able to keep stuffing bytes into the pipe
for the duration of a full time slice without being scheduled away for
Emacs again.  So the low latency of Linux and the quasi-synchronous
(instead of timeslice-based) context switch on select starve the
writing process on single-CPU systems and consequently cause Emacs to
get most of the stuff only in small packets.

So what can we do about this?

a) use only SMP systems
b) pester Linux developers to be less eager with context switches on
   select.  I am trying this course now, but it will of course take
   time to register and other OS might have similar problems
c) make a super-efficient path for process output arriving in tiny
   chunks, since this is what we will be force-fed most of the time.

> > > What happens if you try the following version, which forces
> > > start-process to use a pipe rather than a pty:
> > 
> > Nothing much, I said already.
> 
> Well, it could have an effect for your ACUTeX case, since it would
> then be writing to a pipe (using buffered i/o) rather than to a pty
> (typically using non-buffered i/o).

I already said that the scheduling seems to be pretty much the same.
The pipe never fills up.

> Come on yourself...  This illustrates that if you use
>         "tex ... | dd bs=1k"
> instead of just
>         "tex ..."
> in acutex, you'll probably end up with much faster performance
> since you will get buffered output from tex.

Possibly.  But when we get some prompt output and a feedback is
expected, dd will not flush.

> But you should get the same effect by setting
> process-connection-type to nil.

No, since a pipe does not wait for being full before switching on the
reader.  dd, in contrast, will not write anything out before a read
of 1k has been satisfied.

> > The dd was explicitly there to force small writes, since that is
> > the most common situation when talking with applications through a
> > pipe.
> 
> I would expect that for a pty, not for a pipe!

As I explained, it does not make a noticeable difference for a pipe (I
tried) since the moment something gets into the pipe, the writing
process is starved of any CPU time needed for letting the pipe get
any fuller.

> > And that is what makes Emacs dog slow when compared with XEmacs
> > when running things like TeX from within AUCTeX.
> 
> Maybe Xemacs is using pipes rather than ptys by default (or never
> use pty's at all)...

Pipes don't play into it.  Either it does something other than
select, or its very_small_input_chunk_size path is much more
efficient than Emacs'.

> > We are having here an operating system with a single CPU, and we
> > switch processes here for every byte mainly, and the producing process
> > gets no CPU time for making new bytes while the consuming process is
> > still busy processing the byte it got.  Anyhow, I _do_ have the
> > additional suspicion that there is pretty much one "tick" of delay for
> > every of byte involved and that the CPU is mostly idling for that
> > reason: a 600MHz PIII should even with the full processing overhead
> > due to catering for every single character on its own be able to
> > handle more than 100 characters per second.

Should, but I now have better statistics.

> Could you try
> 
>         M-: (setq process-connection-type nil)
> 
> before you run your acutex process, and see if that makes a
> difference.

Since you refuse to believe me...

pty:

Function Name       Call Count  Elapsed Time  Average Time
==================  ==========  ============  ============
TeX-command-filter  725         0.1004559999  0.0001385599


pipe:

Function Name       Call Count  Elapsed Time  Average Time
==================  ==========  ============  ============
TeX-command-filter  684         0.0818009999  0.0001195921


dd obs=16k & pty

Function Name       Call Count  Elapsed Time  Average Time
==================  ==========  ============  ============
TeX-command-filter  196         0.0215799999  0.0001101020


dd obs=16k & pipe

Function Name       Call Count  Elapsed Time  Average Time
==================  ==========  ============  ============
TeX-command-filter  180         0.0209769999  0.0001165388


And that does not take the context switch time and a lot of other
stuff you can see scroll by into account.  As you can see, the
difference pty/pipe is pretty much irrelevant.  The blocking makes the
difference (the difference is quite more prominent than what appears
here).

Here is my program polished somewhat: you can run it either with
M-x make-test RET
or with

emacs -batch -q -no-site-file -l testio.el -eval "(progn(make-test t 'kill-emacs)(sit-for 100000))"

from the command line.

The relevant part of the output will be something like:
finished
Time:  0
 641 blocks with size    1
   1 blocks with size 1023
   3 blocks with size 1024
Time:  1
 640 blocks with size    1
Time:  2
 449 blocks with size    1
   1 blocks with size 1023
   3 blocks with size 1024
Time:  3
1827 blocks with size    1
Time:  4
1132 blocks with size    1
   1 blocks with size  881
   6 blocks with size 1023
  20 blocks with size 1024


[-- Attachment #2: I/O benchmark --]
[-- Type: application/emacs-lisp, Size: 1557 bytes --]

[-- Attachment #3: Type: text/plain, Size: 422 bytes --]


Note that on multiprocessor machines, the bad effect does not occur.
This is really a single processor problem, and for those operating
systems that do an immediate context switch from a task writing to a
pty or pipe to a task having a select system call on it.

At the very least, we need a well-optimized path for small process
output sizes to deal with that problem.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: Any idea about what makes Emacs slow reading on pipes?

[Kim F. Storm]

David.Kastrup@t-online.de (David Kastrup) writes:

> No, that isn't it.  I can pipe the stuff into |cat >/dev/null instead
> of Emacs, and it will be finished in a whiffy.  Emacs must be using
> some sort of system call that _stops_ dd most of the time from being
> able to so another write to the pipe until the Linux scheduler has run
> again (which it does at the rate of about 100 per second, almost
> exactly the rate at which characters arrive).  

That is because emacs is using a pty by default for process i/o (I
guess that is to avoid buffering output from the writer, so emacs will
get a more steady flow of input to process, e.g. for compiler output),
and it sets the pty to non-blocking mode when reading from the pty 
(to avoid having emacs hanging on the read system call until it has
successfully read all 1024 bytes).

The combined effect is that the write (such as tex) will use
un-buffered writes, and emacs will use non-blocing reads, effectively
using as small data units between writer and reader as possible.

If you just do your-command | cat > /dev/null, your-command will
most likely do buffered i/o, so it will surely run faster.

>                                                I am not saying that
> the current Linux kernel might not play its part in the catastrophic
> performance here, but the fact is that substituting different readers
> for Emacs on the delivering pipe yields in vast performance gains of
> the _writing_ process.
> 
> > What happens if you try the following version, which forces
> > start-process to use a pipe rather than a pty:
> 
> Nothing much, I said already.

Well, it could have an effect for your ACUTeX case, since it would
then be writing to a pipe (using buffered i/o) rather than to a pty
(typically using non-buffered i/o).

> > But if you really want to speed up things, use a block size
> > of 1k like this:
> 
> > 	       "-c" "hexdump -v /dev/zero|dd bs=1k count=100")
> 
> Oh, come on.  

Come on yourself...  This illustrates that if you use
        "tex ... | dd bs=1k"
instead of just
        "tex ..."
in acutex, you'll probably end up with much faster performance
since you will get buffered output from tex.

But you should get the same effect by setting process-connection-type
to nil.


>               The dd was explicitly there to force small writes,
> since that is the most common situation when talking with
> applications through a pipe.  

I would expect that for a pty, not for a pipe!

>                               And that is what makes Emacs dog slow
> when compared with XEmacs when running things like TeX from within
> AUCTeX.

Maybe Xemacs is using pipes rather than ptys by default (or never use
pty's at all)...

> 
> We are having here an operating system with a single CPU, and we
> switch processes here for every byte mainly, and the producing process
> gets no CPU time for making new bytes while the consuming process is
> still busy processing the byte it got.  Anyhow, I _do_ have the
> additional suspicion that there is pretty much one "tick" of delay for
> every of byte involved and that the CPU is mostly idling for that
> reason: a 600MHz PIII should even with the full processing overhead
> due to catering for every single character on its own be able to
> handle more than 100 characters per second.

Could you try

        M-: (setq process-connection-type nil)

before you run your acutex process, and see if that makes a difference.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Any idea about what makes Emacs slow reading on pipes?

[Stefan Monnier]

> Actually, this isn't it.  Pipes behave just the same, and I now know
> why.  You won't see this problem on an SMP machine.  The writing

Indeed my old 266Mhz PII is bi-processor and I don't see it.

> process makes a _small_ write, and Emacs, waiting on the receiving end
> of it with select, gets woken up immediately and gets to process the
> small write.  But while Emacs is processing the small write, the
> writing process does not get any processing time at all, so it won't
> produce anything new until Emacs either goes back to sleep, or is
> preempted because of taking too long (for example garbage collecting).

Sounds pretty convincing.

> So what can we do about this?
> 
> a) use only SMP systems
> b) pester Linux developers to be less eager with context switches on
>    select.  I am trying this course now, but it will of course take
>    time to register and other OS might have similar problems
> c) make a super-efficient path for process output arriving in tiny
>    chunks, since this is what we will be force-fed most of the time.

The problem with (c) is that we can't cut many corners for
this special case, since we have no idea whether some other
output is about to come or not, so we have to do the full
processing, just in case.

I think the problem is really a kernel issue, and it even sounds
like something very vaguely familiar (as in "some kind of textbook
example about why you don't want to systematically switch when
an event wakes up another process").  I haven't hung out in kernel
world for a long time, but I suspect that the problem you describe
depends on the kernel version.  I would also consider it as a
kernel-scheduler bug, since the `dd' process ends up starved.

I'd appreciate if you could CC me on any email you might send
to the kernel crowd.


	Stefan

Re: Any idea about what makes Emacs slow reading on pipes?

[Kai Großjohann]

David.Kastrup@t-online.de (David Kastrup) writes:

> So what can we do about this?
>
> a) use only SMP systems
> b) pester Linux developers to be less eager with context switches on
>    select.  I am trying this course now, but it will of course take
>    time to register and other OS might have similar problems
> c) make a super-efficient path for process output arriving in tiny
>    chunks, since this is what we will be force-fed most of the time.

I wonder if the invocation of the process filter function could be
optimized.

Idea: the current situation might be this: when Emacs gets stuff from
a process, it reads the data, stashes it somewhere in a buffer, then
invokes the process filter function on the stashed-away stuff.

We might change it so that, after stashing the stuff in a buffer,
Emacs looks again if more data is available from the same process.
If so, also read it and stash it in the buffer, too.

That way, Emacs might continue reading stuff from the process until
the process has nothing to produce.

(I think I'm lucky that Tramp doesn't use process filters :-)
-- 
This line is not blank.

Re: Any idea about what makes Emacs slow reading on pipes?

[Kai Großjohann]

Another thought that I'm having is that I wonder whether your kernel
includes the scheduler changes that are supposed to favor interactive
applications.  It seems that the kernel might think that Emacs is an
interactive application, and so it schedules Emacs quickly, hoping to
improve interactive performance.

Just a wild stab in the dark.
-- 
This line is not blank.

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Another thought that I'm having is that I wonder whether your kernel
> includes the scheduler changes that are supposed to favor
> interactive applications.  It seems that the kernel might think that
> Emacs is an interactive application, and so it schedules Emacs
> quickly, hoping to improve interactive performance.
> 
> Just a wild stab in the dark.

It pretty obviously does.  Here is a test: start an xterm.  Then run

od -v /dev/zero|dd obs=1

in it.  Then, in some other shell window, type

while true;do :; done

The original xterm grinds to a halt immediately while the shell is
using up its interactivity bonus.  Then the xterm resumes work again,
at a much _higher_ speed than before.  The moment you kill off the
busy loop, the xterm drops to a crawl again.

I remember this "slow start" xterm effect where an xterm got much
faster real time performance on a busy rather than on an idle system
and performance got good only when the xterm could not keep up, from
a lot of single CPU Unices.  One of those quirks one does not think
about much.

And the faster an operating system can switch contexts, and the more
it prouds itself of little latency, the worse the effects get.

Yes, I hope to persuade the Linux kernel crowd of a scheduling policy
that will fix this situation.  But since Emacs runs on a wagonload of
different systems, and this situation is all too common, it would not
do harm to think of a scheme where Emacs does not spend so much time
on every context switch it is handed out generously.

That is, I would like a scheme where Emacs throttles the filter
routine calls for the benefit of other tasks it is not even
responsible for.  It could maintain its own buffer that it fills
on wakeup without further processing until it is either full or until
a certain time since the last processing has elapsed, and only then
do the much more expensive operation of running a filter routine
instead of yielding again to the operating system.

That is, if the operating system is lavish with incurring context
switches on a single CPU system, it may be because it can switch
contexts quite fast.  So we should let Emacs deal with the context
switch fast, too.  Throttling multibyte conversions and filter
routines with the help of a separate buffer should be a way to
achieve that without impacting performance on systems with a smarter
scheduler.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:
> 
> > So what can we do about this?
> >
> > a) use only SMP systems
> > b) pester Linux developers to be less eager with context switches on
> >    select.  I am trying this course now, but it will of course take
> >    time to register and other OS might have similar problems
> > c) make a super-efficient path for process output arriving in tiny
> >    chunks, since this is what we will be force-fed most of the time.
> 
> I wonder if the invocation of the process filter function could be
> optimized.
> 
> Idea: the current situation might be this: when Emacs gets stuff from
> a process, it reads the data, stashes it somewhere in a buffer, then
> invokes the process filter function on the stashed-away stuff.
> 
> We might change it so that, after stashing the stuff in a buffer,
> Emacs looks again if more data is available from the same process.

The data will not become available unless the process gets the CPU
time to produce it, and we can't persuade the operating system to
give the process the CPU time unless we yield the CPU.  The solution,
as explained elsewhere, is to throttle the calls to the process
filter function and actively delay processing if we find ourselves
processing small packets at a high rate.

> That way, Emacs might continue reading stuff from the process until
> the process has nothing to produce.

Well, people always complain about Emacs being an operating system,
not an editor, but scheduling processes is not yet its responsibility.
I just propose that we help out the operating system in its scheduling
a bit...  since the bad consequences occur on our side of the
equation, we should do what we can reasonable do to avoid them even
where, strictly speaking, the OS is to blame for assuming that Emacs
will be happier with many small packets instead of a few large ones.

But the "slow start xterm" effect, where the performance of an xterm
with

od -v /dev/zero|dd obs=1

gets much faster if you start

while true;do :;done

in a separate term is very common on single-processor machines.  I
always found it curious and never thought about it further.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Kai Großjohann]

David.Kastrup@t-online.de (David Kastrup) writes:

> It pretty obviously does.  Here is a test: start an xterm.  Then run
>
> od -v /dev/zero|dd obs=1
>
> in it.  Then, in some other shell window, type
>
> while true;do :; done

Interestingly enough, I can't observe this effect.  At least, the
output scrolls by really fast in both cases, and maybe I just can't
see the difference between two really fast speeds...

I'm running Linux 2.4.20 on a 2GHz Celeron.  I think that means I
don't have Hyperthreading and therefore just one CPU.

Hm.

I've now looked again on another machine (2GHz Mobile Pentium 4, same
kernel), and I can see the leading digits couting, and they count at
the same speed regardless of the while loop.
-- 
This line is not blank.

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:
> 
> > It pretty obviously does.  Here is a test: start an xterm.  Then run
> >
> > od -v /dev/zero|dd obs=1
> >
> > in it.  Then, in some other shell window, type
> >
> > while true;do :; done
> 
> Interestingly enough, I can't observe this effect.  At least, the
> output scrolls by really fast in both cases, and maybe I just can't
> see the difference between two really fast speeds...

Interestingly enough, I can't reliably reproduce this effect either.
Hilarious.  Maybe it has something to do with what xterms I started as
background processes (different nicety?).  Oh, if I have a block
cursor blinking in yet another window, this seems to help.  Crazy.

> I'm running Linux 2.4.20 on a 2GHz Celeron.  I think that means I
> don't have Hyperthreading and therefore just one CPU.

Perhaps your CPU is to fast.

> I've now looked again on another machine (2GHz Mobile Pentium 4, same
> kernel), and I can see the leading digits couting, and they count at
> the same speed regardless of the while loop.

Well, having just half the computing power available at least should
not leave you with the same speed, should it?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Kai Großjohann]

David.Kastrup@t-online.de (David Kastrup) writes:

> Interestingly enough, I can't reliably reproduce this effect either.
> Hilarious.  Maybe it has something to do with what xterms I started as
> background processes (different nicety?).  Oh, if I have a block
> cursor blinking in yet another window, this seems to help.  Crazy.

I tried "emacs -q -no-site-file" for the blinking cursor.  No dice.
Here's what I did:

I start the dd thing, then click on the Emacs to give it focus.  The
output from dd shortly (for a second or so) slows down, then resumes
again at the previous speed (before Emacs was started).  I then start
the while look which doesn't change the speed, either.  I again click
on the Emacs window to resume the blinking cursor, which causes a
short pause in the dd output, as before.

>> I'm running Linux 2.4.20 on a 2GHz Celeron.  I think that means I
>> don't have Hyperthreading and therefore just one CPU.
>
> Perhaps your CPU is to fast.

Could be.

>> I've now looked again on another machine (2GHz Mobile Pentium 4, same
>> kernel), and I can see the leading digits couting, and they count at
>> the same speed regardless of the while loop.
>
> Well, having just half the computing power available at least should
> not leave you with the same speed, should it?

So maybe neither the dd nor the infloop require 100% CPU.  Maybe the
terminal (KDE program "konsole") is too slow in printing all this
stuff?
-- 
This line is not blank.

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:
> 
> > Interestingly enough, I can't reliably reproduce this effect either.
> > Hilarious.  Maybe it has something to do with what xterms I started as
> > background processes (different nicety?).  Oh, if I have a block
> > cursor blinking in yet another window, this seems to help.  Crazy.
> 
> So maybe neither the dd nor the infloop require 100% CPU.  Maybe the
> terminal (KDE program "konsole") is too slow in printing all this
> stuff?

Sigh.  What exactly about the word "xterm" don't you understand?  If
you don't use a terminal program that gets more efficient as it is
able to process larger batches, the exercise is pointless.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Richard Stallman]

    That is because emacs is using a pty by default for process i/o (I
    guess that is to avoid buffering output from the writer, so emacs will
    get a more steady flow of input to process, e.g. for compiler output),

I am not sure any more what the context is, but the usual reason for
using a pty is so that process groups and associated signals work.

Re: Any idea about what makes Emacs slow reading on pipes?

[Richard Stallman]

    We might change it so that, after stashing the stuff in a buffer,
    Emacs looks again if more data is available from the same process.
    If so, also read it and stash it in the buffer, too.

That is worth trying--it might not be too hard to write.  But I think
the problem is in Linux, so the real solution is to change Linux.

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     We might change it so that, after stashing the stuff in a
>     buffer, Emacs looks again if more data is available from the
>     same process.

More data will not be available from the same process, since the
process has not had any chance to get CPU time again for generating
more data.  We are talking about a uniprocessor machine here.  The
only way in which more data could become available is if Emacs
_voluntarily_ yielded the CPU instead of processing what is there.
So the question is how to yield the CPU, and for how long, and when
to stop.

The best possibility would be to resume when either a number of
characters is available (a "read" of a certain size can be satisfied),
or a timeout has occured.  Unfortunately, I don't see how to achieve
this without actually reading the pipe out all of the time.  So we do
need a temporary buffer which we serve _fast_.  If it is full, we
process it, if not, we may wait.  If waiting does not yield new
characters, fire up the filter.  How long to wait, depending on the
past history and the fill level of our buffer?  Good question.

>     If so, also read it and stash it in the buffer, too.
> 
> That is worth trying--it might not be too hard to write.  But I
> think the problem is in Linux, so the real solution is to change
> Linux.

I am following this way by discussing this on the Linux kernel list
and am pretty certain that a future version of Linux might be able to
deal with this gracefully.  But there are other operating systems of
which I believe a majority to be afflicted with this
uniprocessor-specific problem, and older Linux systems will remain
with us for a long time.  It would be good if we could equip Emacs
with a few heuristics that would not cause noticeable degradations on
operating systems that would know how to deal with this scheduling
anomaly, but improved matters on the _many_ systems that don't.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[Kai Großjohann]

David.Kastrup@t-online.de (David Kastrup) writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> David.Kastrup@t-online.de (David Kastrup) writes:
>> 
>> > Interestingly enough, I can't reliably reproduce this effect either.
>> > Hilarious.  Maybe it has something to do with what xterms I started as
>> > background processes (different nicety?).  Oh, if I have a block
>> > cursor blinking in yet another window, this seems to help.  Crazy.
>> 
>> So maybe neither the dd nor the infloop require 100% CPU.  Maybe the
>> terminal (KDE program "konsole") is too slow in printing all this
>> stuff?
>
> Sigh.  What exactly about the word "xterm" don't you understand?  If
> you don't use a terminal program that gets more efficient as it is
> able to process larger batches, the exercise is pointless.

I apologize.  Four messages ago you mentioned "some other shell
window", and I somehow misread this to mean that the terminal running
dd is not important.

So I now retried the whole thing with two xterms, one running the dd,
and another running the infloop.  I also had a third window running,
a Konsole with a blinking cursor.

I see the same behavior as previously, when I was using two Konsole
windows and Emacs with a blinking cursor.

Hm.
-- 
This line is not blank.

Re: Any idea about what makes Emacs slow reading on pipes?

[Kai Großjohann]

David.Kastrup@t-online.de (David Kastrup) writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     We might change it so that, after stashing the stuff in a
>>     buffer, Emacs looks again if more data is available from the
>>     same process.
>
> More data will not be available from the same process, since the
> process has not had any chance to get CPU time again for generating
> more data.  We are talking about a uniprocessor machine here.

I'm not familiar with the internals of Unix (-like) systems, but maybe
it is possible to "look" in such a way that the scheduler has a chance
to do its thing.  For example, say that using select(2) causes the
scheduler to run, then we might "look" using select(2).

Hm.  But this might make Emacs quite a bit slower.  So maybe it is a
good idea to only look twice if little data has been received.  If a
lot of data has been read, then we can just process that immediately.
-- 
This line is not blank.

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> So I now retried the whole thing with two xterms, one running the
> dd, and another running the infloop.  I also had a third window
> running, a Konsole with a blinking cursor.
> 
> I see the same behavior as previously, when I was using two Konsole
> windows and Emacs with a blinking cursor.
> 
> Hm.

Well, I can't reliably trigger the behavior with xterms here myself.
Probably the empty loop does only in some cases cause preemption of
the xterm, and there was something else involved.

Which kernel, by the way?

Anyhow, I also posted an Emacs test program, and that would probably
pretty reliably show the effect.

My Emacs test program runs about a factor of 3 faster when I have an
od -v /dev/zero xterm running at the same time.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Any idea about what makes Emacs slow reading on pipes?

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> David.Kastrup@t-online.de (David Kastrup) writes:
> 
> > Richard Stallman <rms@gnu.org> writes:
> >
> >>     We might change it so that, after stashing the stuff in a
> >>     buffer, Emacs looks again if more data is available from the
> >>     same process.
> >
> > More data will not be available from the same process, since the
> > process has not had any chance to get CPU time again for
> > generating more data.  We are talking about a uniprocessor machine
> > here.
> 
> I'm not familiar with the internals of Unix (-like) systems, but
> maybe it is possible to "look" in such a way that the scheduler has
> a chance to do its thing.  For example, say that using select(2)
> causes the scheduler to run, then we might "look" using select(2).

The scheduler _has_ already made its decision, and the decision is to
have Emacs run rather than the output generating process.

There are various reasons: Emacs has been waiting on that pipe for a
longer time, so it is judged an "interactive program" and not a
calculating one.  It gets an interactivity bonus and gets scheduled
immediately when available.

This interactivity bonus gets lower while busy processing continues.
That is the reason that after a while, a few larger packets make it
through.

> Hm.  But this might make Emacs quite a bit slower.  So maybe it is a
> good idea to only look twice if little data has been received.  If a
> lot of data has been read, then we can just process that
> immediately.

The only way to yield is to actually and voluntarily call a delay.
This should only be done when there is reason to suspect it might be
worth a try, namely when we find ourselves repeatedly working on
small packets in fast succession.  Emacs should just refuse to call a
process filter too fast twice in a row unless it has stuffed the
buffer sufficiently, and it should have a very fast path when it
decides not to run the process filter yet.

Perhaps a better scheme would be to use _blocking_ reads with the full
buffer length and a timeout instead when something is available.  When
the blocking read fails, get just what is there.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum