From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Zelphir Kaltstahl Newsgroups: gmane.lisp.guile.user Subject: Re: Guile fibers return values Date: Sun, 5 Jan 2020 19:22:14 +0100 Message-ID: <547672e4-30f7-2783-f3d5-199417a71c98@posteo.de> References: <20200105123329.5019662bdf1895a4164faf62@gmail.com> <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="256543"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 Cc: Guile User To: Chris Vine Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Sun Jan 05 19:22:58 2020 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1ioAYE-0014Zo-3Z for guile-user@m.gmane.org; Sun, 05 Jan 2020 19:22:58 +0100 Original-Received: from localhost ([::1]:44226 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ioAYC-0007n8-95 for guile-user@m.gmane.org; Sun, 05 Jan 2020 13:22:56 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40588) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ioAXj-0007cj-NF for guile-user@gnu.org; Sun, 05 Jan 2020 13:22:29 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ioAXi-0000Se-1t for guile-user@gnu.org; Sun, 05 Jan 2020 13:22:27 -0500 Original-Received: from mout02.posteo.de ([185.67.36.66]:38871) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ioAXh-0000Qa-FO for guile-user@gnu.org; Sun, 05 Jan 2020 13:22:25 -0500 Original-Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 80E402400FE for ; Sun, 5 Jan 2020 19:22:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017; t=1578248543; bh=I7rHOHHPj6Kg29n4w/ky7P98mzjyaurdhwJ9dkkkqZQ=; h=Subject:To:Cc:From:Date:From; b=i8FCCQWP1CQDgofKbmfZtI9D3/KMsK0tRLkTpoYq8M4484yv0bErFYUBse4GNCTJr AQNGytGeywTBuqQJbdOzOH6Vxjup2RPgj8kjOQ8q6nhDg5BmCJZBY+D2EdSOXkopmD SXlygOww9JCRxmRVD/AI0GsK8SeQO9bVAP40fqLNMK1h8dZPqWEujtSdhh6rRCNkUE EKBWNrAa5bmwXKMUN4COFNQQ5HDUTINMCmaS34C9bFa+qjB3CoGRrLo5E8pmMEFfTz oKhHcRml159XrCg3P5rzhlbeNxtPCVeUsUtGjx4pPprd/JbeATHnY7mhTspR+Y3iGN PwAepd4ARojOg== Original-Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 47rRnM0brsz9rxj; Sun, 5 Jan 2020 19:22:14 +0100 (CET) In-Reply-To: <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com> Content-Language: en-US X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 185.67.36.66 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:15994 Archived-At: Hi Chris! On 1/5/20 3:23 PM, Chris Vine wrote: > On Sun, 5 Jan 2020 13:58:24 +0100 > Zelphir Kaltstahl wrote: >> Thank you for the detailed explanation! >> >> By "process" I meant only "sequence of steps performed", the main thun= k >> in run-fibers, separate from the steps that are run in the spawned >> fiber, not really OS process or thread. >> >> I will take a look again at the parallel forms and think about whether= I >> want to use them or fibers. >> >> Originally I had my algorithm in Racket and could not get it to work i= n >> parallel, unless I explore places and serializable lambdas more. >> >> I think fibers are more flexible than the parallel forms though, as on= e >> could also build a pipeline using fibers or any kind of network of >> connected computation tasks, while the parallel forms split a task >> immediately and then join again. Not sure any of the additional >> flexibility of fibers helps me. Perhaps I can use both and abstract fr= om >> it with an additional abstraction layer. Then my code could also be us= ed >> more easily in other Schemes. > Fibers are more flexible than guile's parallel forms as you say (of > course, if you have your own thread pool available that can also be > more flexible than the parallel forms), but fibers' main other feature > is that they will multi-plex i/o operations on the same native thread > using guile's suspendable ports and coroutines. Any one native OS > thread run by the fibers' scheduler may therefore have more than one > coroutine running on it. So, beware having blocking operations when > using fibers, because between fibers running on the same native OS > thread, concurrency is co-operative - coroutine task switching occurs o= n > a port suspending or the fiber yielding, although there is some work > stealing implemented between OS threads which will help you out. (From > the blocking point of view, do not use the simple-format procedure as i= n > my toy example because it is not suspendable-port-safe.) > > If your work is i/o-bound rather than cpu-bound, fibers are likely to b= e > especially useful. I think the decision tree calculations, which I want to parallelize, are not I/O related. However, I am not quite sure I understand the whole suspendable port thing, but here is what I think it is: When some I/O happens in a fiber, the fiber is still able to pause (suspend, yield, =E2=80=A6) at that point, because the suspendable ports = work in such a way, that no harm is done in doing so. Later the I/O work can be resumed (woken up from suspension). This quality had to be built into Guile first, before fibers were able to take advantage of it. Is this correct? But I do not understand, why this is not the case with normal OS threads. Maybe it can be done but is not convenient or difficult to get right, to work with suspendable ports, when not using the fibers library? And why is simple-format not "suspendable-port-safe"? (What does a procedure need to do, in order to be "suspendable-port-safe"?) >> This is my project: >> https://notabug.org/ZelphirKaltstahl/guile-ml/src/wip-port-to-guile >> >> I still am not sure though, if I can simply use any lambda I want and >> send that to a fiber, or I need to look out for things like "What is i= n >> the environment of the lambda?". It would be good to know that. I gues= s >> it depends on how data sent on channels is handled in the fibers libra= ry. > The lambda closures passed to run-fibers and spawn-fiber are like any > other lambda closure. In particular, beware of closures with shared > mutable state. Apart from that you should be fine as regards closures > but I am not convinced that I sufficiently understand your use case to > comment further. On data sent through channels, I haven't looked at it > but I have always assumed that objects such as lists, vectors, records > and so forth which are passed through channels are passed by reference > in the ordinary way, so don't mutate them after putting them in a > channel if more than one fiber may subsequently have them in use > concurrently. The purpose of channels is to provide safe > synchronization. Good advice, thanks. I was not planning to do mutation on the things received from channels in a fiber, but it is good to remember, that it might be problematic to do so, even with vectors or structs. So I better create new structs or vectors, when creating a return message. With the parallel forms, isn't it the case, that at each call of such a form, new OS threads are created? In this case it might be a good idea to create a fibers scheduler and reuse it, if that is possible, so that I do not need to create my own process pool kind of thingy. Regards, Zelphir