From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Tom Tromey Newsgroups: gmane.emacs.devel Subject: Re: advice needed for multi-threading patch Date: Fri, 18 Sep 2009 16:59:37 -0600 Message-ID: References: Reply-To: Tom Tromey NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1253314802 8687 80.91.229.12 (18 Sep 2009 23:00:02 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 18 Sep 2009 23:00:02 +0000 (UTC) Cc: Emacs development discussions To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sat Sep 19 00:59:55 2009 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1MomQj-00054G-HA for ged-emacs-devel@m.gmane.org; Sat, 19 Sep 2009 00:59:53 +0200 Original-Received: from localhost ([127.0.0.1]:60224 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MomQi-0000oU-PM for ged-emacs-devel@m.gmane.org; Fri, 18 Sep 2009 18:59:52 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MomQd-0000m4-Gi for emacs-devel@gnu.org; Fri, 18 Sep 2009 18:59:47 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MomQY-0000bm-PK for emacs-devel@gnu.org; Fri, 18 Sep 2009 18:59:47 -0400 Original-Received: from [199.232.76.173] (port=41377 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MomQY-0000bZ-Gs for emacs-devel@gnu.org; Fri, 18 Sep 2009 18:59:42 -0400 Original-Received: from mx1.redhat.com ([209.132.183.28]:35717) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1MomQX-0006pY-UN for emacs-devel@gnu.org; Fri, 18 Sep 2009 18:59:42 -0400 Original-Received: from int-mx03.intmail.prod.int.phx2.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id n8IMxegY025441; Fri, 18 Sep 2009 18:59:40 -0400 Original-Received: from ns3.rdu.redhat.com (ns3.rdu.redhat.com [10.11.255.199]) by int-mx03.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id n8IMxc3n032383; Fri, 18 Sep 2009 18:59:38 -0400 Original-Received: from opsy.redhat.com (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by ns3.rdu.redhat.com (8.13.8/8.13.8) with ESMTP id n8IMxcii001309; Fri, 18 Sep 2009 18:59:38 -0400 Original-Received: by opsy.redhat.com (Postfix, from userid 500) id CA3223782A5; Fri, 18 Sep 2009 16:59:37 -0600 (MDT) X-Attribution: Tom In-Reply-To: (Stefan Monnier's message of "Wed, 26 Aug 2009 15:18:06 -0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) X-Scanned-By: MIMEDefang 2.67 on 10.5.11.16 X-detected-operating-system: by monty-python.gnu.org: Genre and OS details not recognized. X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:115472 Archived-At: Tom> I'm trying to properly implement per-thread let binding in the presence Tom> of buffer-local variables. Stefan> The problem with that is that we first need to figure out what its Stefan> semantics should be. And to me the semantics of a let-bound Stefan> buffer-local variable is very much unclear in the presence of threads Stefan> (it's already pretty messy even without threads). Stefan> So I'd rather disallow other threads from accessing this buffer's Stefan> variable while the let-binding is active. I thought I'd post a status report on the multi-threading patch. First, thanks to Giuseppe, it is all hosted on Gitorious: http://gitorious.org/emacs-mt 'mt' is the branch I've been working on, he's done some additional fixes and changes on g-exp. (I am behind on pulling from him... and we don't agree about everything, hence two branches :-) What works: * You can start a new thread with run-in-thread. The exact use differs between the branches. Threads are cooperative, you can yield with an explicit call to yield. * The GC handles multiple threads. * Many, but not all, of the 9 kinds of variables that Emacs implements work properly with threads. In particular, working ones are defvaralias, objfwd, and ordinary. These all should do the right thing in all cases. Buffer-local and buffer objfwd mostly work, because we do buffer locking. (However, I think make-local-variable doesn't work properly with let-binding and multiple threads.) Not working on intfwd, boolfwd, keyboard-local and frame-local. * Buffer locking. There are also a number of problem areas. * Buffer locking is actually quite a pain. If you have a background thread locking some buffer, you can't switch to that buffer. Emacs will just pause. I don't think we allow interrupting a lock acquisition (clearly a bug). * Giuseppe implemented minibuffer exclusion. But really the keyboard should only be available to one thread at a time. * I think there are some V* variables that probably should always be per-thread. My notes say Vsignaling_function, but I know I've run into other ones. (What is noteworthy about this is that it implies that a big audit of the C code is necessary..) * I/O is also an unsolved problem. Right now it is easy to make emacs appear to hang by running a process filter in one thread while a different thread owns the buffer lock. (Probably, if a thread owns a buffer lock, then only that thread should be able to run associated process filters.) I think Giuseppe probably knows of more problems. Also, there are tons more problems if you want preemptive multithreading... redisplay probably needs work, regex is not reentrant, the GC needs a bit more work, you need lisp-level locks, etc. Sometimes I wonder whether it would be better to just fork a second emacs and communicate with it using pipes + princ + read. This uses more memory but the implementation would be a lot simpler and (I think) it would suffice for the one case that seems most important: Gnus. Anyway, give it a try. We welcome your advice and input. Finally, we've been doing all our discussions privately on irc. I'd prefer we discuss everything in the open. Is it ok to use emacs-devel for this? If not we'll set up a new mailing list somewhere. Tom