From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Chris Vine Newsgroups: gmane.lisp.guile.devel,gmane.lisp.guile.user Subject: Re: anyone define port types? Date: Sun, 19 Jun 2016 16:33:27 +0100 Message-ID: <20160619163327.36246706@dell.homenet> References: <87y492mnjp.fsf@pobox.com> <87pots9tag.fsf@gnu.org> <87bn37wtf2.fsf@pobox.com> <20160612092513.3eb1c8a3@laptop.homenet> <8760t5mthu.fsf@pobox.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1466350455 12019 80.91.229.3 (19 Jun 2016 15:34:15 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 19 Jun 2016 15:34:15 +0000 (UTC) To: guile-user , guile-devel Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Sun Jun 19 17:34:07 2016 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1bEeje-000505-NZ for guile-devel@m.gmane.org; Sun, 19 Jun 2016 17:34:07 +0200 Original-Received: from localhost ([::1]:39086 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEejd-0004sH-N6 for guile-devel@m.gmane.org; Sun, 19 Jun 2016 11:34:05 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:43233) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEejC-0004rO-57 for guile-devel@gnu.org; Sun, 19 Jun 2016 11:33:39 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bEej6-0001YF-SK for guile-devel@gnu.org; Sun, 19 Jun 2016 11:33:37 -0400 Original-Received: from avasout08.plus.net ([212.159.14.20]:42658) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bEej6-0001Xt-IS for guile-devel@gnu.org; Sun, 19 Jun 2016 11:33:32 -0400 Original-Received: from dell.homenet ([87.114.218.169]) by avasout08 with smtp id 8fZT1t0053fstn301fZUww; Sun, 19 Jun 2016 16:33:29 +0100 X-CM-Score: 0.00 X-CNFS-Analysis: v=2.1 cv=O6PEx0JW c=1 sm=1 tr=0 a=4ynVEcF4YjHkqINdvT3UIQ==:117 a=4ynVEcF4YjHkqINdvT3UIQ==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kj9zAlcOel0A:10 a=pD_ry4oyNxEA:10 a=ybZZDoGAAAAA:8 a=Fwo3Pw8wAAAA:8 a=mDV3o1hIAAAA:8 a=NEAV23lmAAAA:8 a=EbfUVmjmAiIiRmr46toA:9 a=B_i6TodRMCCaRQbp:21 a=EHcyv0ymFXa79BK6:21 a=CjuIK1q_8ugA:10 a=0d70bcxeP8kA:10 a=0RhZnL1DYvcuLYC8JZ5M:22 a=sTfVWhjbJpNfc6ZWP1zY:22 a=_FVE-zBwftR9WsbkzFJk:22 a=Bn2pgwyD2vrAyMmN8A2t:22 Original-Received: from dell.homenet (localhost [127.0.0.1]) by dell.homenet (Postfix) with ESMTP id 7BF86443386; Sun, 19 Jun 2016 16:33:27 +0100 (BST) In-Reply-To: <8760t5mthu.fsf@pobox.com> X-Mailer: Claws Mail 3.13.1 (GTK+ 2.24.30; x86_64-unknown-linux-gnu) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x (no timestamps) [generic] X-Received-From: 212.159.14.20 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.org gmane.lisp.guile.devel:18349 gmane.lisp.guile.user:12652 Archived-At: On Sun, 19 Jun 2016 11:13:17 +0200 Andy Wingo wrote: > Hi :) > > On Sun 12 Jun 2016 10:25, Chris Vine > writes: > > >> http://www.gnu.org/software/guile/docs/master/guile.html/Input-and-Output.html > > > > The documentation indicates that with the C ports implementation in > > guile-2.2, reads will block on non-blocking file descriptors. > > Correct. > > > This will stop the approach to asynchronicity used in 8sync and > > guile-a-sync (the latter of which I have written) from working > > correctly with sockets on linux operating systems, because at > > present both of these use guile's wrapper for select. > > The trouble is that AFAIU there is no way to make non-blocking input > work reliably with O_NONBLOCK file descriptors in the approach that > Guile has always used. > > As you know, the current behavior for Guile 2.0 is to throw an > exception when you get EAGAIN / EWOULDBLOCK. If I am understanding > you correctly, your approach is to only read from a port if you have > done a select() / poll() / etc on it beforehand indicating that you > can read at least one byte. My approach would be to to do that, if it worked. And it does work with pipes and unix domain sockets provided you only read a byte at a time (see further below), but does not work on linux with TCP sockets because linux's select() and poll() system calls are not POSIX compliant. Therefore with TCP sockets on linux you have to use non-blocking reads and cater for the possibility of an EAGAIN/EWOULDBLOCK exception. > The problem with this is not only spurious wakeups, as you note, but > also buffering. Throwing an exception when reading in Guile 2.0 will > discard input buffers in many cases. Likewise when writing, you won't > be able to know how much you've written. > > This goes not only for the explicit bufffers attached to ports and > which you can control with `setvbuf', but also implicit buffers, and > it's in this case that it's particularly pernicious: if you > `read-char' on a UTF-8 port, you might end up using local variables > in the stack as a buffer for reconstructing that codepoint. If you > throw an exception in the middle, you discard those bytes. Likewise > for writing. I recognise this problem. The answer I have adopted when reading from TCP sockets is to extract individual bytes only from the port into a bytevector using R6RS's get-u8 procedure and (if the port is textual rather than binary) to reconstruct characters from that using bytevector->string at, say, a line end[1]. An EAGAIN/EWOULDBLOCK exception is then just treated as an invitation to return to the prompt, and read state is retained in the bytevector. In effect this is doing by hand what a more complete non-blocking EAGAIN-safe port implementation might otherwise do for you. Writing is something else. To do it effectively the writer to the port must in any event cater for the fact that when the buffer is full but the underlying file descriptor is ready for writing, the next write will cause a buffer flush, and if the size of the buffer is greater than the number of characters that the file can receive without blocking, blocking might still occur. You usually need to switch off buffering for writes (but you quite often may want to do that anyway on output ports for sockets). > For suspendable ports, you don't throw an exception: you just assume > the operation is going to work, but if you get EAGAIN / EWOULDBLOCK, > you call the current-read-waiter / current-write-waiter and when that > returns retry the operation. Since it operates on the lowest level of > bytes, it's reliable. Looping handles the spurious wakeup case. > > > However, to cater for other asynchronous implementations of file > > watches, would it be possible to provide a configurable option > > either to retain the guile-2.0 behaviour in such cases (which is to > > throw a system-error with errno set to EAGAIN or EWOULDBLOCK), or > > to provide a non-blocking alternative whereby the read operation > > would, instead of blocking, return some special value such as an > > EAGAIN symbol? Either would enable user code then to resume to its > > prompt and let other code execute. > > Why not just (install-suspendable-ports!) and > > (parameterize ((current-read-waiter my-read-waiter)) ...) > > etc? It is entirely possible with Guile 2.1.3 to build an > asynchronous coroutine-style concurrent system in user-space using > these primitives. See the wip-ethread branch for an example > implementation. I would want to continue using an external event loop implemented with poll() or select() and delimited continuations. This makes it relatively trivial to implement asynchronous timeouts and single-threaded task multiplexing (albeit co-operative rather than pre-emptive) as well as file operations, and would also enable the glib event loop to be used for programs which happen to use guile-gnome (although guile-gnome has other issues at present). I don't think I have got to grips with how to do that with read-waiter, because the read-waiter comprises in effect another loop (in which the main event loop with its own prompts would have to run) until the read request has been satisfied. I would need to think about it. Since ethreads use a poll()/epoll() loop, presumably you think it is straightforward enough to integrate the two, even if at present I don't. Writing custom C ports which do what I want with non-blocking descriptors is another option, but I would hope that could be avoided: at that point one would probably use some other solution entirely. On a side issue, I am still trying to understand the point of causing guile-2.2's read of a non-blocking C port to block. The whole point of making a descriptor non-blocking is that that shouldn't happen, and there are circumstances where pealing individual bytes off a non-blocking port as they become available is what you want to do. It makes guile's select wrapper unusable with TCP sockets on linux. I understand that suspendable-ports work differently, but that is another matter. Chris [1] A sample implementation can be seen at https://github.com/ChrisVine/guile-a-sync/blob/master/a-sync/event-loop.scm from line 900 on, and elsewhere in that file.