From mboxrd@z Thu Jan 1 00:00:00 1970 Path: main.gmane.org!not-for-mail From: David.Kastrup@t-online.de (David Kastrup) Newsgroups: gmane.emacs.devel Subject: Re: Fix for slow process output processing (please test). Date: 16 Dec 2003 04:34:04 +0100 Sender: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Message-ID: References: NNTP-Posting-Host: deer.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1071550272 24451 80.91.224.253 (16 Dec 2003 04:51:12 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Tue, 16 Dec 2003 04:51:12 +0000 (UTC) Cc: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Tue Dec 16 05:51:09 2003 Return-path: Original-Received: from quimby.gnus.org ([80.91.224.244]) by deer.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 1AW7Ar-0000tE-00 for ; Tue, 16 Dec 2003 05:51:09 +0100 Original-Received: from monty-python.gnu.org ([199.232.76.173]) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1AW7Aq-0004EK-00 for ; Tue, 16 Dec 2003 05:51:08 +0100 Original-Received: from localhost ([127.0.0.1] helo=monty-python.gnu.org) by monty-python.gnu.org with esmtp (Exim 4.24) id 1AW86t-00061V-JD for emacs-devel@quimby.gnus.org; Tue, 16 Dec 2003 00:51:07 -0500 Original-Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24) id 1AW86i-0005yY-Bu for emacs-devel@gnu.org; Tue, 16 Dec 2003 00:50:56 -0500 Original-Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24) id 1AW86B-0005sz-QJ for emacs-devel@gnu.org; Tue, 16 Dec 2003 00:50:54 -0500 Original-Received: from [194.25.134.81] (helo=mailout03.aul.t-online.de) by monty-python.gnu.org with esmtp (Exim 4.24) id 1AW86B-0005sj-94 for emacs-devel@gnu.org; Tue, 16 Dec 2003 00:50:23 -0500 Original-Received: from fwd08.aul.t-online.de by mailout03.aul.t-online.de with smtp id 1AW78g-0006xM-02; Tue, 16 Dec 2003 05:48:54 +0100 Original-Received: from localhost.localdomain (ZwefkqZSreZTc6EL3ZTg0QjebQLKjUh1FJdhtwIYB7d8x-CB6uzM0L@[217.80.160.99]) by fwd08.sul.t-online.com with esmtp id 1AW78W-26tpS40; Tue, 16 Dec 2003 05:48:44 +0100 Original-Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by localhost.localdomain (8.12.8/8.12.8) with ESMTP id hBG4mhTQ002464 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 16 Dec 2003 05:48:43 +0100 Original-Received: (from dak@localhost) by localhost.localdomain (8.12.8/8.12.8/Submit) id hBG3Y5ui002279; Tue, 16 Dec 2003 04:34:05 +0100 Original-To: no-spam@cua.dk (Kim F. Storm) In-Reply-To: Original-Lines: 47 User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3.50 X-Seen: false X-ID: ZwefkqZSreZTc6EL3ZTg0QjebQLKjUh1FJdhtwIYB7d8x-CB6uzM0L X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.2 Precedence: list List-Id: Emacs development discussions. List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Xref: main.gmane.org gmane.emacs.devel:18709 X-Report-Spam: http://spam.gmane.org/gmane.emacs.devel:18709 no-spam@cua.dk (Kim F. Storm) writes: > David Kastrup and others have reported very slow processing of process > output, particularly with the Linux kernel's scheduler giving emacs > too much attention while starving the processing producing the output. > > Below is a patch which introduces a small delay on reading output from > such processes, without using global delays in processing other events > (this is done by temporarily removing the process' file descriptor from > the call to select and use a short timeout on the select instead. I have glanced over that patch. There is one problem I see with it: if indeed the generating process is able to produce lots of data with a single write call, Emacs will be throttling its input without a purpose. This situation is quite more probable when we have a multiprocessor machine or when the input is generated by some network process (so again, on a different CPU). The best behavior would be a) don't bother reading from the pipe for a while if it is not yet full. b) process it immediately once it is completely full. The select call does not offer any possibility to say "wake up immediately if there is such an amount of data". And we need the select call to poll other input, so we can't just use blocking I/O with a timeout set (which would probably be the best solution if we had only to worry about a single source of input). `Enterprise level' solutions would probably be: a) use blocking I/O with a timeout set in a separate thread. When the timeout hits or the read completes, whatever has accumulated in the corresponding input buffer is marked for processing and made available to the main Emacs. b) use aio_read to request a full pipe worth of data. If no signal announces the availability of the data, some timeout may be used to cancel the aio_read if there is data available, but not a full pipe worth of it. Of course, those solutions are more operating system specific than the current one. It may well be that there are few applications that will turn out worse by throttling Emacs to something like 8kB/20ms or so. -- David Kastrup, Kriemhildstr. 15, 44793 Bochum