From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id IFYUNEXqQmLmnQAAgWs5BA (envelope-from ) for ; Tue, 29 Mar 2022 13:15:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id gBTVMUXqQmIYRwEA9RJhRA (envelope-from ) for ; Tue, 29 Mar 2022 13:15:17 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7E604125D2 for ; Tue, 29 Mar 2022 13:15:17 +0200 (CEST) Received: from localhost ([::1]:50544 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZ9oi-00082C-Jh for larch@yhetil.org; Tue, 29 Mar 2022 07:15:16 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZ9l6-0006b9-Uj for guix-devel@gnu.org; Tue, 29 Mar 2022 07:11:34 -0400 Received: from new3-smtp.messagingengine.com ([66.111.4.229]:54871) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZ9l4-0007L8-90; Tue, 29 Mar 2022 07:11:32 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailnew.nyi.internal (Postfix) with ESMTP id 5B5065800E5; Tue, 29 Mar 2022 07:11:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 29 Mar 2022 07:11:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= philipmcgrath.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=anqhvFC7uzQ2EmvEH66Bdi0SJUwTFBtis+Ifbi MavPs=; b=oK33DYTuRx71zL3oev3ipB1xmO2z2XwpBjzcCNOQUY3QXQQHdudEOB 2K7zY+2wbmDAw2fDWRRZzqtG55GNirVy1SVX+1qTscytMhekBOW0b1AtpBRM8l5O ewGsOTbExrryeeTGbUI8gJwMeAlPVBqEXwpRwyIxonHJW8UuwWf5Pl0W/Kxwn0lq IXQmPjf/gEfqmmDBWK7jXcm49Y5mmXOaoYevBPdl8thYAne3YxWjqHeXCem4PjGn UuyQ/Z78eWyhaFcVdxYQTck/q9Cu51Graw4mIXF5NU5R5v1yvAOCjNp4HJhCQqDz wYIfFFbN2XWitPFMvxwSPmVkoje6ebCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=anqhvFC7uzQ2EmvEH66Bdi0SJUwTFBtis+IfbiMav Ps=; b=fB7oY8hNoa3Clay67Y0kIoYLqO9jXL8RS4s1zmz1lpfMZejSENfVmw7Rs oHQ1wknNZqV2+2LvlrxB6jQiJKde+XJ29VmlSaF0pAnEiq/ortX4c5RRcRjOkoRx c8fFbiraifa11ihTE9bo+quDRk33krnjLIC/evDFRzmg2B19oQX48QhtvLbu/IEy 0wsdv6gQW1WZ/F9NI4xW6viuZQRy6mpz6j6csUwU6jhkOSctVBle0E6aRDYgTJxy 4p9lAMW3y1kGRyLZz9mYmcTERHe5VWMjPNk0bL3nhWOHO/Y9rn2xSZgyqfDguYrP 5dch/rPVaKLVHNFfBFC1Q8Re13p5A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudeitddguddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne goufhushhpvggtthffohhmrghinhculdegledmnecujfgurhepkfffgggfuffvfhfhjggt gfesthekredttdefjeenucfhrhhomheprfhhihhlihhpucfotgfirhgrthhhuceophhhih hlihhpsehphhhilhhiphhmtghgrhgrthhhrdgtohhmqeenucggtffrrghtthgvrhhnpeel gfeiuedvfedvudejvdejueeijeefjeeviefhjedvkeefgfeiffduhffhudefudenucffoh hmrghinhepghhoohhglhgvrdgtohhmpdhuthgrhhdrvgguuhdprhgrtghkvghtqdhlrghn ghdrohhrghdpghhithhhuhgsrdgtohhmpdhmihgtrhhoshhofhhtrdgtohhmpdgrrhgthh hivhgvrdhorhhgpdhnvghurdgvughunecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhhihhlihhpsehphhhilhhiphhmtghgrhgrthhhrdgtoh hm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 29 Mar 2022 07:11:26 -0400 (EDT) Message-ID: <4129627e-da0a-d73a-c754-0a9df765cc01@philipmcgrath.com> Date: Tue, 29 Mar 2022 07:11:25 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: The Shepherd on Fibers Content-Language: en-US To: Maxime Devos , =?UTF-8?Q?Ludovic_Court=c3=a8s?= , guix-devel@gnu.org References: <87ee2sfg9d.fsf@inria.fr> <4f889372-3464-92df-b350-b7c61e081f31@philipmcgrath.com> <9c53fc57b6e4d81b5b4447b5ed2d0797668a70ff.camel@telenet.be> From: Philip McGrath In-Reply-To: <9c53fc57b6e4d81b5b4447b5ed2d0797668a70ff.camel@telenet.be> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=66.111.4.229; envelope-from=philip@philipmcgrath.com; helo=new3-smtp.messagingengine.com X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URI_DOTEDU=1.246 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1648552517; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=anqhvFC7uzQ2EmvEH66Bdi0SJUwTFBtis+IfbiMavPs=; b=TBIspbBsGlKyEUHh1JsQGA2hNs3SmkaS8v4RXYMyFjZhjeRNBBA+MkYQquwzZfFEtJxFWk G/X2uOzdu+Li2mHZCzsJb7NlfrpflkUsjdD5K9q9si4iZ3g9+OAzdZ92UG7h2+dVywhEZJ uPBZeVnRjdiJVOoCUhjVNEzaoQcaCyH4aLfhP8J4NgbvuUqnhBeu38KyupPLKXJHsXVqhw /wb0TeCSeyNBDPTLLKAtMWaVtF1fMPN1DEV8KqLFGd+Z2w3k1IhuNDF9HseePnf/uV7cSL MS+gKD9EdrZLSVZ15nHhuikJRxte/IYmZrOI8BsDEYC5BKltztwe111HjeJz0A== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1648552517; a=rsa-sha256; cv=none; b=eSncEtkAQfQXzepimlhR1zDRh7HW7f5OuobkQhHBhlvX4LhTJsCvHHDYUir2iDi/BlNRT2 lQI0+98Jq8Py5yYfTH2JXfslj7kleSlLIvDXWuKPE+mhDFXXHd0CSzfAqYzDHhHcCpqZNQ D/MP1DDVu1/FZ70zhRO+m5KcuMlwWFvMM8EKEjXJu4PDgxJehO3WccQrRJTgmh+qtCPrSG VTQWXLACxUanX5Po14dpPVsqHP1c5/VSDqtbl3CNpKA+JJbyBN9I1n9oocgXNYlfKxt9Tn A7cazts/zOOR4tDNyNf8V8jKX/Go6xOICmDrxV3KPhwarhZvFsMei209GHWJ2g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm1 header.b=oK33DYTu; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm3 header.b=fB7oY8hN; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 0.53 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm1 header.b=oK33DYTu; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm3 header.b=fB7oY8hN; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 7E604125D2 X-Spam-Score: 0.53 X-Migadu-Scanner: scn0.migadu.com X-TUID: HVeKHJFDJ//n Hi, On 3/29/22 05:36, Maxime Devos wrote: > Philip McGrath schreef op ma 28-03-2022 om 20:14 [-0400]: >> Maybe it would be enough for this case for Fibers to provide >> variants of `dynamic-wind` and/or `call-with-continuation-barrier` >> that cooperate with the Fibers implementation. I don't know if >> that would be possible of not—in addition to my limited familiarity >> with Fibers, I have not read all of the related literature that >> Alexis, Matthew, and Matthias Felleisen discuss in [5] and >> [6]—again, I am not an expert! > > Fibers' context switching is based on continuations. By definition, > ‘call-with-continuation-barrier’ must create a continuation barrier > (and as a consequence, interfere with Fibers' context switching). > > It can be important to let 'call-with-continuation-barrier' (*) > actually create a continuation barrier, even when fibers is used, > e.g. to not create deadlocks (or livelocks or whatever, I don't know > the terminology) when POSIX mutexes are used. As such, as-is, I > don't think so. > > [...] > > (*) Actually, 'call-with-continuation-barrier' and 'dynamic-wind' are > already a bit of a lie, since the kernel ignores them when context > switching ... Maybe continuation barriers and {un,re}winding is a > matter of perspective? Yes, that's the crux of the problem. All of the references I know of are discussed in mailing list threads [1] and [2], plus the more recent Flatt & Dybvig, "Compiler and Runtime Support for Continuation Marks" (PLDI 2020) [3], which discusses the Racket-on-Chez implementation of delimited control. In [1], Matthew Flatt wrote: > For an implementation that relies on a representation choice instead > of tracing or analysis, Racket CS's implementation of delimited > control is the state of the art --- mostly by building on Chez > Scheme's implementation of continuations. Again, I am very far from an expert, but I'll try to distill the relevant parts here. To quote from the abstract and introduction of "Adding Delimited and Composable Control to a Production Programming Environment" [4] (internal citations omitted), > Operators for delimiting control and for capturing composable > continuations litter the landscape of theoretical programming > language research. Numerous papers explain their advantages, how > the operators explain each other (or don’t), and other aspects of > the operators’ existence. Production programming languages, however, > do not support these operators, partly because their relationship to > existing and demonstrably useful constructs—such as exceptions and > dynamic binding—remains relatively unexplored. > > [...] > > Due to this semantic interference, simulations of delimited control > do not immediately yield production-quality implementations. For > example, a Scheme library can use `call/cc` to simulate delimited > continuations, but other libraries that use `call/cc` directly or > that use `dynamic-wind` can interfere with the simulation. > > Over the past year, we have integrated a full set of delimited- > control operators within PLT Scheme, ensuring that all of them > interact properly with the rest of the Scheme programming language > as well as pre-existing extensions in PLT Scheme. Specifically, PLT > Scheme’s prompts and composable continuations have a well-defined > and useful interaction with `call/cc`, `dynamic-wind`, dynamic > binding via continuation marks, and exceptions. Racket's Concurrent ML subsystem also falls into that category. The result of this paper is `racket/control` library[5]. To take Racket CS as an example, Chez Scheme doesn't provide delimited continuations or "green" threads/fibers, but it does provide an efficient and powerful implementation of continuations (even including support for `equal?`!). The Racket CS runtime system implementation uses Chez's `call/cc`, `dynamic-wind`, etc. to implement Racket's primitive control operators (from the built-in module '#%kernel). Then, a larger set of control operators can be implemented as a library in terms of the primitives. But, as the above paper says, this means that Chez's `call/cc`, `dynamic-wind`, etc. are *unsafe* from the perspective of Racket's control primitives. From the docs for Racket's `ffi/unsafe/vm` library [6]: > Beware of directly calling a Chez Scheme primitive that uses Chez > Scheme parameters or `dynamic-wind` internally. Note that `eval`, in > particular, is such a primitive. The problem is that Chez Scheme’s > `dynamic-wind` does not automatically cooperate with Racket’s > continuations or threads. To call such primitives, use the > `call-with-system-wind primitive`, which takes a procedure of no > arguments to run in a context that bridges Chez Scheme’s > `dynamic-wind` and Racket continuations and threads. Anything that has the potential to block Racket's scheduler (as opposed to a fiber/thread), like POSIX mutexes, is likewise unsafe and should be encapsulated in a safe abstraction. For more on this, see the docs for `ffi/unsafe/atomic`[7], `ffi/unsafe/schedule`[8], `ffi/unsafe/port`[9], `ffi/unsafe/os-thread`[10], `ffi/unsafe/global`[11], and the `#:atomic?`, `#:async-apply`, `#:lock-name`, `#:in-original-plce?`, `#:blocking?`, `#:callback-exns?`, and `#:save-errno` arguments to `_cprocedure`[12], plus the section titled "Threads, Threads, Atomicity, Atomicity, and Atomicity" (yes, there are that many layers!) in the file "racket/src/cs/README.txt" [13] from the main Racket Git repository. The key difference with Guile's Fibers is that it uses continuations at the same layer of abstraction available to other Guile code. By "variants of `dynamic-wind` and/or `call-with-continuation-barrier` that cooperate with the Fibers implementation", I meant maybe Fibers could provide something like `call-with-continuation-barrier/except-for-fibers` and the Shepherd could use it. But I don't know enough about the implementation details to know if that would actually be a viable option. It seems like `dynamic-wind` and reliable resource finalization are particular problems in this respect. Yesterday I started looking at the technical report Alexis King mentioned in [1], “Algebraic Effect Handlers with Resources and Deep Finalization”[14]. (Apparently there is a deep correspondence between algebraic effect handlers and delimited control.) In § 8.7 "Dynamic Wind", it says (internal citations omitted): > The Scheme language always supported delimited continuations and > they have also struggled with initialization- and finalization > for continuations that resumed more than once. The > `unwind-protect` in Scheme is like a `finally` clause, while > `dynamic-wind` is like `initially`/finally with a pre- and > postlude. Sitaram describes how the standard dynamic-wind is not > good enough in general: > >> While this may seem like a natural extension of the first-order >> `unwind-protect` to a higher-order control scenario, it does not >> tackle the pragmatic need that `unwind-protect` addresses, >> namely, the need to ensure that a kind of “clean-up” happens only >> for those jumps that significantly exit the block, and not for >> those that are a minor excursion. The crux is identifying which >> of these two categories a jump falls into. > > Interestingly, this is exactly what is addressed by algebraic > effects where “significant exit”s are operations that do not > resume, while “minor excursions” are regular operations that > resume with a result. (That Sitaram quote is from [15].) This sounds more promising than anything else I've heard of! Enough to make me want to finish reading that paper :) However, I know extremely little about algebraic effects, and I have no idea how this might translate into delimited control in the Scheme tradition, much less Guile or Fibers specifically. I think Alexis might be the most likely person to be able to answer that question. -Philip [1]: https://groups.google.com/g/racket-users/c/AAeEp_llxaY/m/uRviksPGAgAJ [2]: https://groups.google.com/g/racket-dev/c/PyetqHSjtAA/m/slBdazdqAwAJ [3]: https://www.cs.utah.edu/plt/publications/pldi20-fd.pdf [4]: https://www.cs.utah.edu/plt/publications/icfp07-fyff.pdf [5]: https://docs.racket-lang.org/reference/cont.html [6]: https://docs.racket-lang.org/foreign/vm.html [7]: https://docs.racket-lang.org/foreign/Atomic_Execution.html [8]: https://docs.racket-lang.org/foreign/Thread_Scheduling.html [9]: https://docs.racket-lang.org/foreign/Ports.html [10]: https://docs.racket-lang.org/foreign/Operating_System_Threads.html [11]: https://docs.racket-lang.org/foreign/unsafe-global.html [12]: https://docs.racket-lang.org/foreign/foreign_procedures.html [13]: https://github.com/racket/racket/blob/master/racket/src/cs/README.txt [14]: https://www.microsoft.com/en-us/research/uploads/prod/2018/04/resource-v1.pdf [15]: https://web.archive.org/web/20200223020813/http://www.ccs.neu.edu/~dorai/uwcallcc/uwcallcc.html