From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: Concurrency, again (was: Emacs Lisp's future) Date: Mon, 10 Oct 2016 10:17:17 +0300 Message-ID: <83twckekqq.fsf@gnu.org> References: <87wq97i78i.fsf@earlgrey.lan> <86k2dk77w6.fsf@molnjunk.nocrew.org> <9D64B8EA-DB52-413D-AE6A-264416C391F3@iotcl.com> <83int1g0s5.fsf@gnu.org> Reply-To: Eli Zaretskii NNTP-Posting-Host: blaine.gmane.org X-Trace: blaine.gmane.org 1476083878 21444 195.159.176.226 (10 Oct 2016 07:17:58 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 10 Oct 2016 07:17:58 +0000 (UTC) Cc: lars@nocrew.org, toon@iotcl.com, emacs-devel@gnu.org To: John Wiegley Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Oct 10 09:17:52 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1btUqJ-0004F8-M1 for ged-emacs-devel@m.gmane.org; Mon, 10 Oct 2016 09:17:47 +0200 Original-Received: from localhost ([::1]:47877 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btUqH-00070l-Qs for ged-emacs-devel@m.gmane.org; Mon, 10 Oct 2016 03:17:45 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:55940) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btUpk-00070Y-MB for emacs-devel@gnu.org; Mon, 10 Oct 2016 03:17:14 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1btUpg-0004iZ-8i for emacs-devel@gnu.org; Mon, 10 Oct 2016 03:17:12 -0400 Original-Received: from fencepost.gnu.org ([2001:4830:134:3::e]:43929) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btUpg-0004hy-5g; Mon, 10 Oct 2016 03:17:08 -0400 Original-Received: from 84.94.185.246.cable.012.net.il ([84.94.185.246]:1968 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1btUpd-0007kX-IH; Mon, 10 Oct 2016 03:17:06 -0400 In-reply-to: (message from John Wiegley on Sun, 09 Oct 2016 19:59:09 -0700) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:208140 Archived-At: > From: John Wiegley > Cc: Toon Claes , lars@nocrew.org, emacs-devel@gnu.org > Date: Sun, 09 Oct 2016 19:59:09 -0700 > > >>>>> Eli Zaretskii writes: > > > There's already a concurrency branch in the Emacs repository. It needs some > > work before it could be merged to master, so if there are people here who > > wants this badly enough, I suggest that they continue work almost done by > > Tom Tromey, who developed that branch, instead of starting anew. > > I think it needs more than just a little work. I spoke to Tom on the phone > last year, and we both agreed it's not a foregone conclusion that that branch > represents the right way of approaching concurrency for an application like > Emacs. We will never know if we don't try it. We will also never collect the experience we need to define "the right way" of approaching concurrency in Emacs, if what's already on the branch is not good enough. Meanwhile, chances of someone coming aboard with the kind of experience and intimate knowledge of Emacs and concurrency that even approaches that of Tom, and motivated enough to do that kind of development using some other model, are non-existent. IOW, saying that the concurrency branch is not good for us without trying to use it seriously is tantamount to saying that Emacs will never get any concurrency features at all, and all this sub-thread is yet another example of issues which never advance anywhere beyond mere talking about them. That's too bad, IMO, because I think something like that could open opportunities for important new features. > In general, users don't want programmatic concurrency. AFAIK, the concurrency branch offers programmatic concurrency only as an opt-in option. Some concurrency happens automatically without any code Lisp programmers need to write. So I see no problem here. What we need is for creative Lisp programmers to take this model and explore how it could be applied to solving real-life problems, with or without programmatic control of the concurrency, see what are its limitations and advantages, and generally collect the kind of experience we sorely need to advance Emacs in that direction. > I fear we'd be debugging subtle interaction issues for the rest of > our lives if we just merged Tom's branch in today. It implements a Java-style > form of concurrency, dependent on mutexes and locking to achieve coherency, > which is fiendishly difficult to get right, even if you confine its use to > just a few small tasks. The hardest bug I ever debugged in my life was a > concurrency bug; the second hardest wasn't even close (and thankfully, I > didn't try to solve it at the same time). Did you look at the code, though? If not, I suggest you do. The model is very simple, and its implementation is also simple. Suffices it to say that at the time I could port that to MS-Windows and debug the few problems in a matter of few days work. When I finished, the only known problem I left unsolved was the one described here: http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00803.html and in a followup: http://lists.gnu.org/archive/html/emacs-devel/2013-09/msg00009.html For those who want some background, I pointed to long discussions of the related issues here: http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg00068.html > I'd much rather we re-examine the goals we want to achieve with concurrency, > an then ask if there are other ways to get there besides, well, concurrency. > For example, if we had a lightweight forking mechanism with transparent > communication between sub-ordinate processes (think async.el, but in C and > highly efficient), I think we could achieve what users want, without the > downsides developers don't want. Even the popular web browsers are moving in > this direction, since it gives a similar effect to threading, but without as > many of the downsides; or take the popularity of the Actor model, used to > reduce the coherency problem down to just mailboxes. Sorry for being blunt, but without someone actively working on this, this is just talk. Who do we have on board who can implement this in C in a way that will work on more than a single platform? On some important platforms, forking is anything but "lightweight", btw. And if async.el is any evidence, it's fiendishly hard to debug and even understand. E.g., it took me some time to understand where to plug the necessary coding stuff, back when we had problems with debbugs clients related to non-ASCII characters. In general, debugging processing that is distributed between several processes is much harder than debugging a single multi-threaded application. Since Emacs is mostly in maintenance mode, this aspect should IMO be significant in any development decision we make. > The reason I love Haskell for its STM concurrency (software transactional > memory) is that it makes certain classes of problems impossible to express. I > believe we would need a mechanism like that for Emacs Lisp, so no one ever has > to hunt down cyclic mutex locks, or reference counts, or why two operations > that should be atomic aren't. I'd rather have a single-threaded Emacs for a > quite a while longer before inviting these sorts problems into our lives. But we don't even know whether these problems are relevant to what Tom implemented in the concurrency branch. We've gone a long way since this issue was first brought up. We've changed our sources in significant ways to support concurrency: all those BVAR and KVAR macros that are all around the C sources were introduced for that very purpose. Likewise, the globals.h header file, and the fact that each variable exposed to Lisp is a member of some C struct -- all this was for supporting concurrency. The code for this is written, debugged, and is only a little ways from being ready for prime time. I'm sorry, but in these conditions discarding all that work without even giving it a serious try makes very little sense to me. If we indeed don't intend to use the concurrency branch, how about removing all that BVAR/KVAR ballast from our sources, to put our money where our mouth is?