From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Daniel Colascione Newsgroups: gmane.emacs.devel Subject: Re: Concurrency, again Date: Thu, 27 Oct 2016 14:50:06 -0700 Message-ID: <8B50FA4C-3EB4-4F73-B26F-BCBC8D2BFD30@dancol.org> References: <87wq97i78i.fsf@earlgrey.lan> <86k2dk77w6.fsf@molnjunk.nocrew.org> <9D64B8EA-DB52-413D-AE6A-264416C391F3@iotcl.com> <83int1g0s5.fsf@gnu.org> <83twckekqq.fsf@gnu.org> <83funkwfzf.fsf@gnu.org> <87k2cwe4wl.fsf@jupiter.lan> <8360odu2gp.fsf@gnu.org> <87r3717brt.fsf@dustycloud.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1477605051 25424 195.159.176.226 (27 Oct 2016 21:50:51 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Thu, 27 Oct 2016 21:50:51 +0000 (UTC) User-Agent: K-9 Mail for Android Cc: Eli Zaretskii , stefan.huchler@mail.de, Philipp Stephani , emacs-devel@gnu.org To: Christopher Allan Webber Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Oct 27 23:50:46 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bzsZB-0004Dt-Qv for ged-emacs-devel@m.gmane.org; Thu, 27 Oct 2016 23:50:30 +0200 Original-Received: from localhost ([::1]:44760 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzsZE-0001Xq-Bf for ged-emacs-devel@m.gmane.org; Thu, 27 Oct 2016 17:50:32 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:42154) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzsZ3-0001WS-G9 for emacs-devel@gnu.org; Thu, 27 Oct 2016 17:50:22 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bzsZ2-0000iE-Cc for emacs-devel@gnu.org; Thu, 27 Oct 2016 17:50:21 -0400 Original-Received: from dancol.org ([2600:3c01::f03c:91ff:fedf:adf3]:36488) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1bzsZ0-0000fn-Ep; Thu, 27 Oct 2016 17:50:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=dancol.org; s=x; h=Message-ID:CC:To:Date:From:Subject:Content-Type:Content-Transfer-Encoding:MIME-Version:References:In-Reply-To; bh=/p8iB+yRGXdYy5imRXrZIrqlR0tTWnPtgdUaz9Ow6Fg=; b=p9fUvxq+LulyzvDJ9bw1hcd4lpFLQQwbbXtnkDSTKIUTDHe0J48UeYMQBhq/LUF7sqf/bWWQkgZlSBvm4lO7Oe4BVOsWzOxgtPk3BFWRftkNhBfm8QmygQeKnyFMJ1SAPibglcmVCDKmjiHUdLbZ5gFmcn8GzK+l5nnC5QFX4pttOEGAcKZtRSUP2K1tYa5JcSM/8LzOjHowWuG8VBU+Krt9yxURPIG7GXLuiSVPC2TtWLk7xb9ErDUeTQvmp6M1mRTe1pDybJYA+K3NX33ImCr2PXl86g318Qhv/S+ZP2Fd8agOhowshE6iBa18xvbuudaatK/Hv0aDGQ0BOVp+rw==; Original-Received: from [2620:0:1008:fd00:9d52:2914:f60f:a6f7] by dancol.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1bzsYy-0000M4-Ah; Thu, 27 Oct 2016 14:50:16 -0700 In-Reply-To: <87r3717brt.fsf@dustycloud.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2600:3c01::f03c:91ff:fedf:adf3 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:208898 Archived-At: On October 27, 2016 1:55:02 PM PDT, Christopher Allan Webber wrote: >Daniel Colascione writes: > >> On 10/27/2016 10:27 AM, Eli Zaretskii wrote: >>>> From: Philipp Stephani >>>> Date: Tue, 25 Oct 2016 23:28:51 +0000 >>>> >>>> I've pushed the experimental branch to 'concurrency-libtask'. It's >essentially a simple wrapper around libtask, >>>> which implements CSP based on setcontext. I've also implemented a >Windows equivalent based on >>>> Windows native fibers, but haven't tried that yet. >>> >>> Thanks. >>> >>> Could you perhaps summarize the relative advantages and >disadvantages >>> of the two concurrency branches? Your branch doesn't have any >>> documentation besides doc strings of the new primitives, so it's not >>> easy to grasp the high-level picture by looking at the details. >>> >>> One issue that bothers me is whether it's wise to use libtask here, >>> because the changes you did there seem to imply that we will have to >>> maintain the library (which is pretty low-level stuff) as part of >>> Emacs. Isn't using system threads better? >> >> Agreed on system threads vs libtask. Fibers of the sort libtask >provides >> (similar to GNU Pth) have some claimed advantages in efficiency in >large >> programs with lots of concurrent tasks, but for us, I vote for >> Python-style use of OS threads with a Python-style GIL that we can >> release to do long-running computations and recover from parallelism. > >Python's GIL + threading is probably not the best model to work off >of... while the GIL is largely misunderstood, there have been a lot of >challenges with getting this model to work nice. At one point, it >turned out that the GIL + threads would result in constant CPU >thrashing. This isn't the case any more, but it was once, and was a >challenging thing to fix: > > http://dabeaz.com/python/UnderstandingGIL.pdf > >Even now, my understanding is that Python does a ton of context >switching, and while it's better than it was, you don't really get a >lot >of the performance you would over threads being used in other systems, >because it's hard for Python to really run things concurrently. Thus >the rise of things like asyncio, where instead of parallelizing, you >break things into a lot of coroutines which don't block on any IO. Of course they block on IO, in their own terms. On a different level of abstraction, they're non blocking. They are exactly as blocking as OS's threads, except you're doing the work the OS already does, and probably better, at least at Emacs scale. I favor maintainability and developer familiarity over raw performance. If we're merely as successful as Python, I don't think I'll be terribly unhappy > >Other mechanisms like green threading + coroutines for most IO bound >async stuff and, in the case that you need CPU bound stuff to be >optimized, something that spawns independent processes (or okay, maybe >threads) that communicate with message passing is probably much better >than a GIL + system threads route, I'd think The numpy approach helps for the odd truly compute bound task. You can implement message passing on top of a GIL system. I understand you as claiming that a green thread system is more powerful than a real thread system. Can you see how? From where I'm standing, it has the same limitations *and* it's harder to release the GIL and do something heavy like JPEG decoding.