From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Chris Vine Newsgroups: gmane.lisp.guile.user Subject: Re: Limiting parallelism using futures, parallel forms and fibers Date: Wed, 8 Jan 2020 11:44:02 +0000 Message-ID: <20200108114402.2cabbd011c52456a73a2fca8@gmail.com> References: <04cb0461-18a1-ef17-4db7-2475c7c806e6@posteo.de> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="155276"; mail-complaints-to="usenet@blaine.gmane.org" Cc: Guile User To: Zelphir Kaltstahl Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Wed Jan 08 12:44:24 2020 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1ip9ku-000XRm-Sd for guile-user@m.gmane.org; Wed, 08 Jan 2020 12:44:09 +0100 Original-Received: from localhost ([::1]:42528 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ip9kt-00030s-9A for guile-user@m.gmane.org; Wed, 08 Jan 2020 06:44:07 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35330) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ip9kb-00030l-Eb for guile-user@gnu.org; Wed, 08 Jan 2020 06:43:50 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ip9kX-0002UJ-JR for guile-user@gnu.org; Wed, 08 Jan 2020 06:43:48 -0500 Original-Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]:51898) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ip9kW-0002T5-8z for guile-user@gnu.org; Wed, 08 Jan 2020 06:43:45 -0500 Original-Received: by mail-wm1-x32d.google.com with SMTP id d73so2154393wmd.1 for ; Wed, 08 Jan 2020 03:43:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Vci3VMmb+zB6oj38E04FTrsSUK3fhCJ2rJ5MdgZDdIA=; b=tvPKF69+bw76f/ejOIbTjr7+AkmrCAAa/IolM7G5+CSM4jYw6+IB5+WIHtCf/Z55eI OAF4CBjAuc5nyjzvyOHNvxw0t8vc2Cvu/cpg4dbzikO2fiuMdboGFTaE+ljgf7bHTavW /UIxM2hFQLpIY2uDMKK6FljPzkFYlTFL/VT1mv5FPvhlaMfDMgI1bOlncR6+Icx8KkbA pCd4Rjuzw+ypJwqCfplJHTbRvdQwUMhwpwn+TtaLoFdSWJtpjsIffmsxx8XkA5RJr214 XlJu0kBsbJqApVb92ufkaTe1tENlf25154CYuIgcL0KZuv6GdxnUsn7hQLkKO9uLswDZ nLvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Vci3VMmb+zB6oj38E04FTrsSUK3fhCJ2rJ5MdgZDdIA=; b=m/O4iXZ9gjjfLDVWr3femWh9d6wG3JWsyLUjIFwNESmIVRN++bknxx+EvCUMMYxbEK g3yWGYNI2slfq30vYIGJX0OpF7hq4RcPUK7mLbu5bgoHTWVRx3Ev3DOV5k6fE3uc4S/K P5vpZe9rBlzs10mKKj0rPlx4hmQfj3Rw0qMZXQxHa6K2ZoiykqGIpl5/SSNlC1wnB69C 4gJQXAiiIE5n/o4F8b0ME4hrMVBIbsqD3qK/J6RE35ttQBt81QiCrHsETCknMWWvxy1y JhlMBLupWUwGzbGM+4jXBuJvJKwydZVjnem70c3UUfVz4Ze/EZlIp9hINWGyZNyuuhwt xEsA== X-Gm-Message-State: APjAAAXFjcup3dElEa3zo1k3c0+5Owuz/RHF543D/Ha1YkHpjIO7nhXO 2ZlLv/EmrTFcpX/waxDjK7ljyrIq X-Google-Smtp-Source: APXvYqylknj4fIvUVkKRlEYeDojgAaxtG3I/Qs5RHoKHpEJNx0bcvIAqxgvBJM9w/ajp0HJG1dzfJQ== X-Received: by 2002:a1c:730d:: with SMTP id d13mr3298010wmb.126.1578483821658; Wed, 08 Jan 2020 03:43:41 -0800 (PST) Original-Received: from bother.homenet ([91.110.243.20]) by smtp.gmail.com with ESMTPSA id z123sm3577661wme.18.2020.01.08.03.43.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jan 2020 03:43:41 -0800 (PST) Original-Received: from bother.homenet (localhost [127.0.0.1]) by bother.homenet (Postfix) with SMTP id E689F261251; Wed, 8 Jan 2020 11:44:02 +0000 (GMT) In-Reply-To: <04cb0461-18a1-ef17-4db7-2475c7c806e6@posteo.de> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::32d X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:16009 Archived-At: On Wed, 8 Jan 2020 08:56:11 +0100 Zelphir Kaltstahl wrote: [snip] > So my questions are: > > - Is there a default / recommended way to limit parallelism for > recursive calls to parallel forms? > > - Is there a better way than a global counter with locking, to limit the > number of futures created during recursive calls? I would dislike very > much to have to do something like global state + mutex. > > - What do you recommend in general to solve this? I think you have it wrong, and that futures use a global queue and a global set of worker threads. I don't see how futures could work without at least a global set of worker threads. Have a look at the futures source code. If you want more control over the thread pool than guile's futures provide, you could consider something like this: https://github.com/ChrisVine/guile-a-sync2/blob/master/a-sync/thread-pool.scm But then you would have to make your own futures if you want a graph of futures rather than a graph of coroutines (which is what you would get if you use the associated event loop). The main thing is to get the parallel algorithm right, which can be tricky.