From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.devel Subject: concurrency suggestions for Gnus (was: Patch for fields of `struct buffer') Date: Mon, 07 Feb 2011 13:48:26 -0600 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: <87pqr3vd6d.fsf_-_@lifelogs.com> References: <4D46E75E.7080503@harpegolden.net> <4D47E65E.1030901@gmail.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: dough.gmane.org 1297114998 20044 80.91.229.12 (7 Feb 2011 21:43:18 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 7 Feb 2011 21:43:18 +0000 (UTC) To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Feb 07 22:43:14 2011 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PmYrY-00072u-QI for ged-emacs-devel@m.gmane.org; Mon, 07 Feb 2011 22:43:13 +0100 Original-Received: from localhost ([127.0.0.1]:47896 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PmYb7-0001qM-9X for ged-emacs-devel@m.gmane.org; Mon, 07 Feb 2011 16:26:13 -0500 Original-Received: from [140.186.70.92] (port=42369 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PmYJ8-0000T6-Nd for emacs-devel@gnu.org; Mon, 07 Feb 2011 16:07:40 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PmX4j-0000jM-CG for emacs-devel@gnu.org; Mon, 07 Feb 2011 14:48:42 -0500 Original-Received: from lo.gmane.org ([80.91.229.12]:49818) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PmX4j-0000jI-2c for emacs-devel@gnu.org; Mon, 07 Feb 2011 14:48:41 -0500 Original-Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1PmX4i-0002DY-8u for emacs-devel@gnu.org; Mon, 07 Feb 2011 20:48:40 +0100 Original-Received: from 38.98.147.130 ([38.98.147.130]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 Feb 2011 20:48:40 +0100 Original-Received: from tzz by 38.98.147.130 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 07 Feb 2011 20:48:40 +0100 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 43 Original-X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 38.98.147.130 X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6; d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/24.0.50 (gnu/linux) Cancel-Lock: sha1:JV3YaPrJMdwN9uE4h9bun7KAqgI= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 80.91.229.12 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:135691 Archived-At: On Sun, 06 Feb 2011 19:34:51 -0700 Tom Tromey wrote: Tom> If we went the "lock anything" route, I would suggest a weak hash table Tom> for locks, instead of putting the lock into the object. A bloom filter would guarantee no false negatives, which as you noted is the vast majority of the cases, requires very little space per element (about 16 bytes to ensure max 0.1% false positives), and runs in constant time. So maybe that's a better first-check data structure in this particular case with the hash table as the authoritative check. On Sun, 06 Feb 2011 19:47:51 -0700 Tom Tromey wrote: Tom> My earlier target was gnus, but I think Lars has probably already Tom> rewritten all the slow bits. Surely making it fully async can't be far Tom> behind :-) Here are some Gnus ideas: Rebuilding the thread tree in the summary buffer can probably be parallelized. For large summaries (over 5000 articles) it can be a long wait. The Gnus registry is a way to track messages by message ID. It needs to process completely independent lists of data in parallel (classic map-reduce) all the time, but does it serially right now. It can probably add extra baggage to each list to help this tracking, but that would make the registry less generic and more complicated. So parallelizing that search would be nice. The spam.el library does many spam check/process funcalls. This can be slow when the spam processing backend uses an ELisp function and it's annoyingly complicated when shell commands are used. Splitting incoming mail can generally be parallelized since each message is split independently of the others. This is not true in some cases but the majority should benefit. The auth-source.el library could search multiple backends simultaneously (this refers to a recent rewrite that's currently in a branch in the Gnus Git repo). This parallel search is not possible right now. Ted