From mboxrd@z Thu Jan 1 00:00:00 1970 Path: main.gmane.org!not-for-mail From: Simon Josefsson Newsgroups: gmane.emacs.devel Subject: Re: What's the problem? Date: Wed, 10 Dec 2003 01:58:08 +0100 Sender: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Message-ID: References: <20031117040607.C6C5D79B72@server2.messagingengine.com> <87ekvpx18d.fsf@emptyhost.emptydomain.de> <4nad6cikxy.fsf@holmes.bwh.harvard.edu> <4nllpt3hr3.fsf@lockgroove.bwh.harvard.edu> <5bad69zd43.fsf@lister.roxen.com> <4noeuon378.fsf@lockgroove.bwh.harvard.edu> <4ny8tsgxy6.fsf@lockgroove.bwh.harvard.edu> <4nhe0ggv0u.fsf@lockgroove.bwh.harvard.edu> <4nk75bwjaf.fsf@lockgroove.bwh.harvard.edu> <4nsmjv8d32.fsf@collins.bwh.harvard.edu> Original-Received: from quimby.gnus.org ([80.91.224.244]) by deer.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 1ATshf-0005T3-00 for ; Wed, 10 Dec 2003 01:59:47 +0100 Original-Received: from monty-python.gnu.org ([199.232.76.173]) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1ATshe-0004MA-00 for ; Wed, 10 Dec 2003 01:59:47 +0100 Original-Received: from localhost ([127.0.0.1] helo=monty-python.gnu.org) by monty-python.gnu.org with esmtp (Exim 4.24) id 1ATtev-0000m8-Ia for emacs-devel@quimby.gnus.org; Tue, 09 Dec 2003 21:01:01 -0500 Original-Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24) id 1ATteU-0000hc-N7 for emacs-devel@gnu.org; Tue, 09 Dec 2003 21:00:34 -0500 Original-Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24) id 1ATtdy-0000cG-BS for emacs-devel@gnu.org; Tue, 09 Dec 2003 21:00:33 -0500 Original-Received: from [217.13.230.178] (helo=yxa.extundo.com) by monty-python.gnu.org with esmtp (TLSv1:DES-CBC3-SHA:168) (Exim 4.24) id 1ATtdx-0000bp-Li for emacs-devel@gnu.org; Tue, 09 Dec 2003 21:00:02 -0500 Original-Received: from latte (yxa.extundo.com [217.13.230.178]) (authenticated bits=0) by yxa.extundo.com (8.12.10/8.12.10) with ESMTP id hBA0wVAU027872 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Wed, 10 Dec 2003 01:58:32 +0100 Original-To: Juri Linkov X-Payment: hashcash 1.2 0:031210:juri@jurta.org:7c01e37e11f0807f X-Hashcash: 0:031210:juri@jurta.org:7c01e37e11f0807f X-Payment: hashcash 1.2 0:031210:emacs-devel@gnu.org:4e6e5414276867a1 X-Hashcash: 0:031210:emacs-devel@gnu.org:4e6e5414276867a1 In-Reply-To: <87iskpbloe.fsf@mail.jurta.org> (Juri Linkov's message of "Wed, 10 Dec 2003 01:45:05 +0200") User-Agent: Gnus/5.1003 (Gnus v5.10.3) Emacs/21.3.50 (gnu/linux) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.2 Precedence: list List-Id: Emacs development discussions. List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+emacs-devel=quimby.gnus.org@gnu.org Xref: main.gmane.org gmane.emacs.devel:18594 X-Report-Spam: http://spam.gmane.org/gmane.emacs.devel:18594 Juri Linkov writes: > Simon Josefsson writes: >> Luke Gorrie writes: >>> People are writing concurrent programs in Elisp today. Most programs >>> interacting with external processes and sockets do so without blocking >>> Emacs. Can't we all just do the same? >> >> No. >> >> I believe several examples have been given already, but the most >> prominent I notice frequently is generating summary buffers in Gnus. >> This is CPU bound. Most of the time is NOT spent waiting from network >> IO, even for 64kbps users, if you profile large groups. It is spent >> traversing internal data structures, sorting and filtering the >> headers. If this is done asynchronously, it would still lock up >> Emacs. > > Many CPU bound tasks can be improved by memoizing computation results. > Since most cited here example is generating summary buffers in Gnus, > I want to note that this particular problem can be solved by storing > computed thread trees in files similar to `.marks' and `.overview'. > To keep track of computed results input parameters should be stored > along with computed data. Yes, it can be improved, but you are talking about caching. It does not remove the need to at some point compute the data stored in the cache. Then Gnus would look up, which I find unacceptable. Furthermore, because a lot of what is done during preparing summary buffers is dynamic, or almost so because it is influenced by hundreds of user variables that may change over time, I'm not convinced caching summary buffer data can be done reliable without great pain. For me, the summary buffer look different most time I enter it, because each added or removed article in the group may change large parts of the summary buffer to be recomputed (which I don't consider a bug). It is certainly easier to compute the summary buffer every time it is needed, and use the user variables during that computation, than reading from a cache and figure out what has to be re-computed given the then-current user variables, and then re-compute the data and store it in the cache. >> Another example is getting new mail in Gnus. This is probably more >> network bound than CPU bound, but doing it asynchronously would still >> lock up Emacs completely for many seconds, which is unacceptable. If >> you use Gnus as a offline reader, getting new mail may take minutes. > > Many tasks can be performed by external processes. Getting new mail > shouldn't lock up Emacs when done by an external program like movemail. I meant parsing the new incoming articles and splitting them into the proper group. This is done in elisp. If you profile Gnus with ELP, you see that the amount of time spent waiting for external processes like movemail is not the biggest cause of the lockups. Most of it is spent evaluating elisp. Even when getting new mail. So optimizing the movemail case to be asynchronous would not improve the situation much, but will make it more difficult to debug and troubleshoot Gnus.