Re: anyone define port types?

[Andy Wingo <wingo@pobox.com>]

On Sun 19 Jun 2016 17:33, Chris Vine <chris@cvine.freeserve.co.uk> writes:

> The answer I have adopted when reading from TCP sockets is to extract
> individual bytes only from the port into a bytevector using R6RS's
> get-u8 procedure and (if the port is textual rather than binary) to
> reconstruct characters from that using bytevector->string at, say, a
> line end[1].  An EAGAIN/EWOULDBLOCK exception is then just treated as
> an invitation to return to the prompt, and read state is retained in
> the bytevector.

Yep, makes sense, though it will be really slow of course.  And, you'd
have to make people rewrite their code to use your I/O primitives
instead of using read-line from (ice-9 rdelim); a bit of a drag, but
OK.

>> Why not just (install-suspendable-ports!) and
>> 
>>   (parameterize ((current-read-waiter my-read-waiter)) ...)
>> 
>> etc?  It is entirely possible with Guile 2.1.3 to build an
>> asynchronous coroutine-style concurrent system in user-space using
>> these primitives. See the wip-ethread branch for an example
>> implementation.
>
> I would want to continue using an external event loop implemented with
> poll() or select() and delimited continuations.

Yes yes a million times yes!  That's what I want too.  To that end, the
current-read-waiter parameter's value is a function which takes the port
being waited on, and when it returns the operation will retry.  In
wip-ethreads I suspend to a prompt, add the port's fd to an epoll set,
and when the fd becomes readable again we resume the continuation,
resulting in a return from the current-read-waiter.

> I don't think I have got to grips with how to do that with read-waiter,
> because the read-waiter comprises in effect another loop (in which the
> main event loop with its own prompts would have to run) until the read
> request has been satisfied.  I would need to think about it.  Since
> ethreads use a poll()/epoll() loop, presumably you think it is
> straightforward enough to integrate the two, even if at present I don't.

Here is the code.  First, a helper:

  ;; The AFTER-SUSPEND thunk allows the user to suspend the current
  ;; thread, saving its state, and then perform some other nonlocal
  ;; control flow.
  ;;
  (define* (suspend #:optional (after-suspend (lambda (ctx thread) #f)))
    ((abort-to-prompt (econtext-prompt-tag (current-econtext))
                      after-suspend)))

Assume there is some current-econtext parameter with a "context" which
holds the prompt tag associated with the scheduler.  As you see when the
continuation resumes, it resumes by calling a thunk, allowing exceptions
to be thrown from the context of the suspend.

Here's the main loop function, which you could replace with the GLib
main loop or whatever:

  (define* (run #:optional (ctx (ensure-current-econtext))
                #:key (install-suspendable-ports? #t))
    (when install-suspendable-ports? (install-suspendable-ports!))
    (parameterize ((current-econtext ctx)
                   (current-read-waiter wait-for-readable)
                   (current-write-waiter wait-for-writable))
      (let lp ()
        (run-ethread ctx (next-thread ctx))
        (lp))))

Cool.  Now the wait functions:

  (define (wait-for-readable port)
    (wait-for-events port (port-read-wait-fd port) (logior EPOLLIN EPOLLRDHUP)))

  (define (wait-for-writable port)
    (wait-for-events port (port-write-wait-fd port) EPOLLOUT))

Now the wait-for-events function:

  (define (wait-for-events port fd events)
    (handle-events
     port
     events
     (suspend
      (lambda (ctx thread)
        ...))))

Well that's a mess, but the thing to know is that the `suspend' will
abort to the relevant prompt, and then invoke the thunk that's its
argument.  Here's `handle-events' and we are done:

  (define (handle-events port events revents)
    (unless (zero? (logand revents EPOLLERR))
      (error "error reading from port" port)))

But I guess that error could be "reading or writing"; oh well.

See
http://git.savannah.gnu.org/cgit/guile.git/tree/module/ice-9/ethreads.scm?h=wip-ethreads&id=253dc1a7114b89351a3aa330caf173b98c5a65dd,
it's not long but it takes some time to read.  I think I can fairly ask
that of you though, given your interest in this area :)

> On a side issue, I am still trying to understand the point of causing
> guile-2.2's read of a non-blocking C port to block.  The whole point of
> making a descriptor non-blocking is that that shouldn't happen, and
> there are circumstances where pealing individual bytes off a
> non-blocking port as they become available is what you want to do.  It
> makes guile's select wrapper unusable with TCP sockets on linux.  I
> understand that suspendable-ports work differently, but that is another
> matter.

I would start by saying that many things are possible including adding
the ability to enable the error-throwing behavior.  I think your use
case is a great one and we should support it well.  I think though that
perhaps you didn't see how the waiter parameters could apply to your use
case; are things clearer now?  I think that once you allow yourself to
use port buffers, things will be much easier and much more efficient as
well.  But, I could be misunderstanding too.

Guile's I/O in the past was very oriented towards ASCII.  2.0 went a
long way towards good textual I/O with other encodings, and with 2.2 we
will have good textual non-blocking I/O.  It could be that by focusing
on this, I have neglected some binary non-blocking I/O use cases, though
I think that get-bytevector-some, having enabled suspendable ports,
serves many of these purposes very well.  But if there is a need for an
additional binary I/O primitive, let's figure out what it is and
implement it.

Andy