From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Ken Raeburn Newsgroups: gmane.emacs.devel Subject: Re: Threads in emacs implementation Date: Tue, 7 Jun 2005 01:16:53 -0400 Message-ID: References: <6dbd4d0005060619227dd41364@mail.gmail.com> NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 (Apple Message framework v622) Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Trace: sea.gmane.org 1118121493 22330 80.91.229.2 (7 Jun 2005 05:18:13 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Tue, 7 Jun 2005 05:18:13 +0000 (UTC) Cc: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Jun 07 07:18:12 2005 Return-path: Original-Received: from lists.gnu.org ([199.232.76.165]) by ciao.gmane.org with esmtp (Exim 4.43) id 1DfWTE-0003Kc-Vs for ged-emacs-devel@m.gmane.org; Tue, 07 Jun 2005 07:17:49 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1DfWZg-0007Df-L1 for ged-emacs-devel@m.gmane.org; Tue, 07 Jun 2005 01:24:28 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1DfWYu-00071E-1D for emacs-devel@gnu.org; Tue, 07 Jun 2005 01:23:40 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1DfWYs-00070X-IG for emacs-devel@gnu.org; Tue, 07 Jun 2005 01:23:39 -0400 Original-Received: from [199.232.76.173] (helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1DfWYs-0006zp-4g for emacs-devel@gnu.org; Tue, 07 Jun 2005 01:23:38 -0400 Original-Received: from [207.172.4.63] (helo=smtp04.mrf.mail.rcn.net) by monty-python.gnu.org with esmtp (Exim 4.34) id 1DfWYx-00022i-EI for emacs-devel@gnu.org; Tue, 07 Jun 2005 01:23:45 -0400 Original-Received: from 65-78-24-4.c3-0.smr-ubr1.sbo-smr.ma.cable.rcn.com (HELO raeburn.org) (65.78.24.4) by smtp04.mrf.mail.rcn.net with ESMTP; 07 Jun 2005 01:17:56 -0400 X-IronPort-AV: i="3.93,176,1115006400"; d="scan'208"; a="43970563:sNHT25289260" Original-Received: from [18.101.0.226] (laptop.raeburn.org [18.101.0.226]) by raeburn.org (8.12.11/8.12.11) with ESMTP id j575Gtkv015682; Tue, 7 Jun 2005 01:16:55 -0400 (EDT) In-Reply-To: <6dbd4d0005060619227dd41364@mail.gmail.com> Original-To: Denis Bueno X-Mailer: Apple Mail (2.622) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:38228 X-Report-Spam: http://spam.gmane.org/gmane.emacs.devel:38228 On Jun 6, 2005, at 22:22, Denis Bueno wrote: > Having said that, my question is, has thought been put into > multithreading parts of Emacs? Or exposing a rudimentary threading API > to Emacs Lisp? This could easily devolve into a war about threading > apis, but, just support some simple subset of pthreads would work. I've thought about it, yes. As Miles said, there's a lot of global state, and lots of code that doesn't expect threading issues. Indeed, the basic Lisp system itself would need changes to deal with any sort of preemptive threading system; dynamic bindings are currently implemented by overwriting a symbol's value, and keeping track on the stack of what value needs to be restored later. Garbage collection also gets more interesting when multiple threads are manipulating Lisp objects at a time. (One could go with fully cooperative threads, but then you still have to rewrite all your I/O to be non-blocking, you lose any benefits of multi-threading in third-party library code that might block, can't take any advantage of multiple CPUs, etc.) For various reasons, I decided to put my attention about three years or so ago into a somewhat different project -- making it possible to run Emacs with the Guile interpreter tied into the Lisp system. Guile -- GNU's Ubiquitous Intelligent Language for Extension -- is the GNU Project's Scheme implementation, intended to be a general extension language for the GNU Project. Scheme is Lisp-like in some ways, different in others; in particular, Scheme uses static scoping, so Lisp dynamic bindings are an interesting part of the issue. But there are people working on making Guile be able to handle Emacs Lisp, and there already is pthread support in Guile. So my thinking is, if the Emacs Lisp engine can be made to operate on Guile objects (including, for example, a new Guile object type that represents an Emacs buffer, which is a Lisp type now), and these other people find ways to deal with threads and dynamic scoping, then we've made a lot of progress not just towards the possibility of multiple threads in Emacs, but also towards having GNU's "ubiquitous extension language" available in what's probably GNU's most commonly extended (and most extensible) program. (There would be other effects also, like having one *massive* application stress-testing the heck out of the Guile allocator and GC system. Might be good for Guile in the long run, but I suspect the initial performance might not be as good as what Emacs has already; then again, I might be pleasantly surprised. There's a research project out there on translating Lisp to Scheme and determining when you might be able to optimize out the whole dynamic-binding mess and use local values, which may translate to more efficient code. If Emacs buffers can be made into an independent object type, perhaps other GNU programs can incorporate them via Guile extensions. Guile has compiled regular expressions as their own type; wouldn't it be nice if Gnus article splitting could cache all of the compiled regular expressions it uses even when GC runs, instead of just some number compiled in to search.c?) I started doing some work on isolating the Lisp system from the rest of Emacs -- removing some assumptions here and there about how Lisp objects were constructed, that sort of thing. But then I got sidetracked with pesky things like work, wow, probably a couple of years ago now, and wasn't able to make much progress for a long time. (Unicode branch guys -- I'm sorry I never got that merge to your branch done that was discussed back then, but by the time anyone got back to me saying "yes", I was already too hosed to pay attention.) I've managed to get some more time to put into it recently, but (a) I've changed jobs and need to re-file some of the paperwork for donating code, and (b) there's been talk of some kind of code freeze or at least feature freeze, though I haven't been able to keep up on the mail quite enough to know what the state of things is. So, I'm not sending in any code changes yet. (And haven't got a lot of them yet anyways.) That sort of work is relatively easy (even if sometimes spread through lots of code -- the widespread changes are usually simple and mechanical), and contributes in some ways towards keeping the code cleaner outside of the guts of the Lisp engine itself. Should we go all the way and actually replace the Lisp object representation with something based on Guile? Some people say yes, some people say absolutely not. Some think we should be rewriting Emacs into some other language. Me, I just want to get something that works, maybe fix some bugs in Guile and in Emacs that might be found in the process, and then see where things stand... Even if that happens, I have only put a little thought into what the Emacs Lisp interface for threads would probably look like. Actually, unless someone comes up with a more friendly wrapper package, I suspect the interface would look a lot like the Scheme interface in Guile -- thread creation, mutex locks, condition variables, etc. Plus, I would guess, locks in objects like buffers, windows, and frames to keep them from getting corrupted. Message passing, work queue management, and so forth could be built on top of the primitives without too much difficulty, in either Lisp or Scheme. I think I'd be more concerned about the UI -- we probably don't want two threads vying for user input and stealing characters from one another or something. And, of course, all that global state Miles was talking about... (Note, too, that once you start using pthreads, you get new restrictions on maximum stack depth, things like that; it can't grow unbounded. That translates directly into a limit on the depth of Lisp function calls, not by number, but by C stack frame size.) > The background for my question: In Gnus, certain network operations > often take a long time -- sometimes I don't want to check my email > when it prompts me for a password, but I hit Return anyway, causing > Emacs to visibly hang for a minute or two. Sometimes whatever server > Gnus wants is down (or DNS is down, or whatever), and it 'hangs' until > it's timed out. This downtime is in many cases at least long enough > that I could profitably be doing work in other Emacs buffers. This, in fact, was my motivation for thinking about it. But, I was also doing a little hacking on Guile at the same time, and, well, I'm sure some would say it was all downhill from there.... Ken