From mboxrd@z Thu Jan 1 00:00:00 1970 Path: main.gmane.org!not-for-mail From: storm@cua.dk (Kim F. Storm) Newsgroups: gmane.emacs.devel Subject: Re: Fix for slow process output processing (please test). Date: 16 Dec 2003 11:23:32 +0100 Sender: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Message-ID: References: NNTP-Posting-Host: deer.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1071566714 15983 80.91.224.253 (16 Dec 2003 09:25:14 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Tue, 16 Dec 2003 09:25:14 +0000 (UTC) Cc: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Tue Dec 16 10:25:10 2003 Return-path: Original-Received: from quimby.gnus.org ([80.91.224.244]) by deer.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 1AWBS2-0001SJ-00 for ; Tue, 16 Dec 2003 10:25:10 +0100 Original-Received: from monty-python.gnu.org ([199.232.76.173]) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1AWBS2-00071h-00 for ; Tue, 16 Dec 2003 10:25:10 +0100 Original-Received: from localhost ([127.0.0.1] helo=monty-python.gnu.org) by monty-python.gnu.org with esmtp (Exim 4.24) id 1AWCPB-0002hr-Mp for emacs-devel@quimby.gnus.org; Tue, 16 Dec 2003 05:26:17 -0500 Original-Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24) id 1AWCP7-0002hc-Fp for emacs-devel@gnu.org; Tue, 16 Dec 2003 05:26:13 -0500 Original-Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24) id 1AWCOb-0002WB-8X for emacs-devel@gnu.org; Tue, 16 Dec 2003 05:26:12 -0500 Original-Received: from [212.88.64.25] (helo=mail-relay.sonofon.dk) by monty-python.gnu.org with smtp (Exim 4.24) id 1AWCOa-0002Uz-KY for emacs-devel@gnu.org; Tue, 16 Dec 2003 05:25:40 -0500 Original-Received: (qmail 97812 invoked from network); 16 Dec 2003 09:24:13 -0000 Original-Received: from unknown (HELO kfs-l.imdomain.dk.cua.dk) (213.83.150.2) by 0 with SMTP; 16 Dec 2003 09:24:13 -0000 Original-To: David.Kastrup@t-online.de (David Kastrup) In-Reply-To: Original-Lines: 84 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3.50 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.2 Precedence: list List-Id: Emacs development discussions. List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Xref: main.gmane.org gmane.emacs.devel:18711 X-Report-Spam: http://spam.gmane.org/gmane.emacs.devel:18711 David.Kastrup@t-online.de (David Kastrup) writes: > > David Kastrup and others have reported very slow processing of process > > output, particularly with the Linux kernel's scheduler giving emacs > > too much attention while starving the processing producing the output. > > > > Below is a patch which introduces a small delay on reading output from > > such processes, without using global delays in processing other events > > (this is done by temporarily removing the process' file descriptor from > > the call to select and use a short timeout on the select instead. > > I have glanced over that patch. There is one problem I see with it: > if indeed the generating process is able to produce lots of data with > a single write call, Emacs will be throttling its input without a > purpose. With the patch, emacs only throttles if it reads less than 1024 bytes at a time from the process -- i.e. if the process is unable to produce data fast enough to fill emacs's read buffer. So if you have a process which manages to write a lot of data in one write, emacs should happily read that data with no throttling. Of course, if it has just read some (slower) data with throttling, the first few 1024 packages will still be trottled (max 100ms total), but then it would read the rest at full speed. > > This situation is quite more probable when we have a multiprocessor > machine or when the input is generated by some network process (so > again, on a different CPU). If you have a multi-processor system, and your process generates output at a steady (fast) stream, emacs will not do throttling on that process. Only if that process slows down, you will see throttling -- but how would you notice that? And network traffic is not throttled by this patch! > > The best behavior would be > a) don't bother reading from the pipe for a while if it is not yet > full. > b) process it immediately once it is completely full. c) or process it after some timeout if there is any data in the pipe Yes, there are various ways to accomplish this, but I would rather try to find a simple solution which works reasonably well in most/all cases, rather than a more complex (and perhaps o/s specific) solution that I'm not really sure will work much better in general. > > The select call does not offer any possibility to say "wake up > immediately if there is such an amount of data". And we need the > select call to poll other input, so we can't just use blocking I/O > with a timeout set (which would probably be the best solution if we > had only to worry about a single source of input). > > `Enterprise level' solutions would probably be: > a) use blocking I/O with a timeout set in a separate thread. When > the timeout hits or the read completes, whatever has accumulated in > the corresponding input buffer is marked for processing and made > available to the main Emacs. > b) use aio_read to request a full pipe worth of data. If no signal > announces the availability of the data, some timeout may be used to > cancel the aio_read if there is data available, but not a full pipe > worth of it. > > Of course, those solutions are more operating system specific than the > current one. It may well be that there are few applications that will > turn out worse by throttling Emacs to something like 8kB/20ms or so. > We may go for more advanced solutions, if the current patch does not solve the problem (maybe with some tuning of the throttling parameters). But I would like some feedback on the effects of the current patch first (also on multi-processor systems if possible); maybe it just happens to be "good enough" in practice. -- Kim F. Storm http://www.cua.dk