From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Christopher Allan Webber Newsgroups: gmane.lisp.guile.devel Subject: Re: wip-ports-refactor Date: Wed, 06 Apr 2016 23:16:59 -0500 Message-ID: <87pou282vo.fsf@dustycloud.org> References: <87twjempnf.fsf@pobox.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: ger.gmane.org 1460002937 26223 80.91.229.3 (7 Apr 2016 04:22:17 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 7 Apr 2016 04:22:17 +0000 (UTC) Cc: guile-devel@gnu.org To: Andy Wingo Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Thu Apr 07 06:22:15 2016 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1ao1SP-0000Yj-G3 for guile-devel@m.gmane.org; Thu, 07 Apr 2016 06:22:13 +0200 Original-Received: from localhost ([::1]:47567 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ao1SO-0002OC-Ow for guile-devel@m.gmane.org; Thu, 07 Apr 2016 00:22:12 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:35177) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ao1SI-0002O5-Ca for guile-devel@gnu.org; Thu, 07 Apr 2016 00:22:07 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ao1SH-0007aE-7z for guile-devel@gnu.org; Thu, 07 Apr 2016 00:22:06 -0400 Original-Received: from dustycloud.org ([2600:3c02::f03c:91ff:feae:cb51]:44716) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ao1SH-0007a4-3d for guile-devel@gnu.org; Thu, 07 Apr 2016 00:22:05 -0400 Original-Received: from oolong (localhost [127.0.0.1]) by dustycloud.org (Postfix) with ESMTPS id 0D7B52667C; Thu, 7 Apr 2016 00:22:03 -0400 (EDT) In-reply-to: <87twjempnf.fsf@pobox.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2600:3c02::f03c:91ff:feae:cb51 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:18275 Archived-At: Andy Wingo writes: > Hi, > > I have been working on a refactor to ports. The goal is to have a > better concurrency story. Let me tell that story then get down to the > details. Hoo, what an email! I need to read your code before I can do a full commentary. But... > More appropriate is 8sync, a new project by Chris Webber that is > designed to be a kind of user-space threading library for Guile. Hey, thanks! I hope it's on the right track. > I did give a try at prototyping such a thing a long time ago, > "ethreads". Ethreads are user-space threads, which are really delimited > continuations with a scheduler. If the thread -- the dynamic extent of > a program that runs within a prompt -- if the thread would block on I/O, > it suspends itself, returning to the scheduler, and then the scheduler > resumes the thread when I/O can continue. There's an epoll loop > underneath. > > That hack seemed to work; I even got the web server working on it, and > ran my web site on it for a while. The problem was, though, that it > completely bypassed ports. It made its own port types and buffers and > everything. That's not really Guile -- that's a library. > > * * * > > Which brings us to the port refactor. Ultimately I see ports as all > having buffers. These buffers can be accessed from Scheme. Normal I/O > goes to the buffer first. When the buffers need filling or emptying, > Scheme code can call Scheme code to do that. There could be Scheme > dynamic parameters defining whether filling/emptying blocks -- if it > doesn't block, then if the read would block it could call out to some > context to suspend the thread. Since it's all Scheme code, that > continuation can be resumed as well -- the delimited continuation does > not capture a trampoline through C. The buffer itself is represented as > a bytevector with a couple of cursors, which gives us some basic > threadsafety without locks on the Scheme side -- Scheme always checks > that accesses are within bounds. > > But, currently in Guile 2.0 and in master, buffering is handled by the > port implementation. That means that there is no buffer to expose to > Scheme, and no real path towards implementing efficient I/O operators > that need to grovel in a buffer from Scheme. It also means that there's > no easy solution for non-blocking I/O, AFAIU. > > The wip-port-refactor branch is a step towards centralizing buffering > management within the generic ports code. It thins the interface to > port implementations, instead assuming that the read/write functions are > to unbuffered mutable stores, as Guile is the part handling the > buffering. I've documented what I can in the branch. So, does this branch replace ethreads, or compliment it? Where should I be focusing my (currently limited) review / integration attempt energy? I've been hoping to review ethreads this week but now I'm unsure. Can you explain how the efforts currently relate? > The commits before the HEAD are fairly trivial I think; it's the last > one that's a doozy. It doesn't yet remove locks; there's still a lot of > locks, and it's hard to know what we can do without locks given the > leeway give to modern C compilers. But it's a step. > > Going forward we need to define a Scheme data type for ports, and to > allow the read/write procedures to be called from Scheme, and to allow > Scheme implementaitons of those procedures. We also need to figure out > how to do non-blocking I/O, both on files and non-files; should we set > all our FD's to O_NONBLOCK? How does it affect our internal > interfaces? I do not know yet. One other question is if this will help in the "no nice way to do custom binary ports" stuff that was blocking the tls-enabled-ports-in-guile-proper thing... > There's still space for different schedulers. I wouldn't want to > include a scheduler and a thread concept in Guile 2.2.0 I don't think -- > but if we can build it in such a way that it seems natural, on top of > ports, then it sounds like a good idea. As I've said, I'm not tied to 8sync specifically if doing something more internally makes more sense. (Even if I have a nice site and logo coming together now ;)) Exciting times! - Chris