asynchronous socket library

asynchronous socket library

[Aleix Conchillo Flaqué]

Hi,

I was wondering if there is (or someone is working on) an asynchronous
socket library for Guile. I was thinking in a pure scheme library (not
bindings for libevent or some other).

Thanks,

Aleix

Re: asynchronous socket library

[Nala Ginrut]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

在 2013-7-16 AM7:41，"Aleix Conchillo Flaqué" <aconchillo@gmail.com>写道：
>
> Hi,
>
> I was wondering if there is (or someone is working on) an asynchronous
> socket library for Guile. I was thinking in a pure scheme library (not
> bindings for libevent or some other).
>

That would be good!
What do you plan for? IOCP?

> Thanks,
>
> Aleix
>

[-- Attachment #2: Type: text/html, Size: 561 bytes --]

Re: asynchronous socket library

[Aleix Conchillo Flaqué]

On Mon, Jul 15, 2013 at 6:03 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
>
> 在 2013-7-16 AM7:41，"Aleix Conchillo Flaqué" <aconchillo@gmail.com>写道：
>
>
>>
>> Hi,
>>
>> I was wondering if there is (or someone is working on) an asynchronous
>> socket library for Guile. I was thinking in a pure scheme library (not
>> bindings for libevent or some other).
>>
>
> That would be good!
> What do you plan for? IOCP?
>

No plans yet. I was just wondering if anyone was working on it.

I still have to read more about it, but it might be a fun one. The
idea just came so I could use it in guile-redis to have non-blocking
calls, some of the redis clients are already asynchronous.

Aleix

Re: asynchronous socket library

[Javier Sancho]

Aleix Conchillo Flaqué wrote:
> I was wondering if there is (or someone is working on) an asynchronous
> socket library for Guile. I was thinking in a pure scheme library (not
> bindings for libevent or some other).

If you don't use libevent, at least you'll need poll, epoll or
something similar. Or how are you planning to monitor activity on a
file descriptor?

--
Javier Sancho Fernández - http://www.jsancho.org/
Associate Member of the Free Software Foundation - http://www.fsf.org/
Contra el DRM - http://www.defectivebydesign.org/what_is_drm

Re: asynchronous socket library

[Aleix Conchillo Flaqué]

On Mon, Jul 15, 2013 at 11:15 PM, Javier Sancho <jsf@jsancho.org> wrote:
> Aleix Conchillo Flaqué wrote:
>> I was wondering if there is (or someone is working on) an asynchronous
>> socket library for Guile. I was thinking in a pure scheme library (not
>> bindings for libevent or some other).
>
> If you don't use libevent, at least you'll need poll, epoll or
> something similar. Or how are you planning to monitor activity on a
> file descriptor?
>

My idea was to create something simple on top of system calls, as
libevent does. This might mean that I need additional system calls
that are not in guile yet. But it doesn't mean I need to create
bindings for libevent (which would probably be smarter if I wanted
something complete). Basilica, the main idea is to learn in the
process.

Aleix

Re: asynchronous socket library

[Aleix Conchillo Flaqué]

On Tue, Jul 16, 2013 at 12:02 AM, Aleix Conchillo Flaqué
<aconchillo@gmail.com> wrote:

> something complete). Basilica, the main idea is to learn in the

Oopsss.... spell checker did a funny thing here. I meant "Basically".

Aleix

Re: asynchronous socket library

[Chaos Eternal]

i think there is a nio branch for guile?
nio means non-blocking io

On Tue, Jul 16, 2013 at 3:02 PM, Aleix Conchillo Flaqué
<aconchillo@gmail.com> wrote:
> On Mon, Jul 15, 2013 at 11:15 PM, Javier Sancho <jsf@jsancho.org> wrote:
>> Aleix Conchillo Flaqué wrote:
>>> I was wondering if there is (or someone is working on) an asynchronous
>>> socket library for Guile. I was thinking in a pure scheme library (not
>>> bindings for libevent or some other).
>>
>> If you don't use libevent, at least you'll need poll, epoll or
>> something similar. Or how are you planning to monitor activity on a
>> file descriptor?
>>
>
> My idea was to create something simple on top of system calls, as
> libevent does. This might mean that I need additional system calls
> that are not in guile yet. But it doesn't mean I need to create
> bindings for libevent (which would probably be smarter if I wanted
> something complete). Basilica, the main idea is to learn in the
> process.
>
> Aleix
>

Re: asynchronous socket library

[Aleix Conchillo Flaqué]

On Tue, Jul 16, 2013 at 12:36 AM, Chaos Eternal <chaoseternal@shlug.org> wrote:
> i think there is a nio branch for guile?
> nio means non-blocking io
>

Couldn't find it with that name.

Aleix

Re: asynchronous socket library

[Chaos Eternal]

http://git.savannah.gnu.org/gitweb/?p=guile.git;a=shortlog;h=refs/heads/wip-ethreads

should be this one

On Tue, Jul 16, 2013 at 11:24 PM, Aleix Conchillo Flaqué
<aconchillo@gmail.com> wrote:
> On Tue, Jul 16, 2013 at 12:36 AM, Chaos Eternal <chaoseternal@shlug.org> wrote:
>> i think there is a nio branch for guile?
>> nio means non-blocking io
>>
>
> Couldn't find it with that name.
>
> Aleix

Re: asynchronous socket library

[Aleix Conchillo Flaqué]

On Tue, Jul 16, 2013 at 8:58 AM, Chaos Eternal <chaoseternal@shlug.org> wrote:
> http://git.savannah.gnu.org/gitweb/?p=guile.git;a=shortlog;h=refs/heads/wip-ethreads
>
> should be this one
>

Oh, I see now.

Nala Ginrut actually asked about the status of the wip-ethreads branch
earlier this month.

http://lists.gnu.org/archive/html/guile-devel/2013-07/msg00000.html

No replies yet. I'll have a look at it anyway.

Thanks,

Aleix

Re: asynchronous socket library

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

() Aleix Conchillo Flaqué <aconchillo@gmail.com>
() Tue, 16 Jul 2013 00:02:36 -0700

   My idea was to create something simple on top of system calls,
   as libevent does.

Have you looked at GNU Serveez?

 http://www.gnu.org/software/serveez/

It is relatively simple.

[cc trimmed]
-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: asynchronous socket library

[Aleix Conchillo Flaqué]

On Tue, Jul 16, 2013 at 12:50 AM, Thien-Thi Nguyen <ttn@gnu.org> wrote:
> () Aleix Conchillo Flaqué <aconchillo@gmail.com>
> () Tue, 16 Jul 2013 00:02:36 -0700
>
>    My idea was to create something simple on top of system calls,
>    as libevent does.
>
> Have you looked at GNU Serveez?
>
>  http://www.gnu.org/software/serveez/
>
> It is relatively simple.
>

No, I didn't know about it, thanks. The other idea of the original
email was to gather some information. I think I have enough to keep me
busy for a while.

Thanks,

Aleix

Re: asynchronous socket library

[mark]

Aleix Conchillo Flaqué <aconchillo@gmail.com> writes:

> Hi,
>
> I was wondering if there is (or someone is working on) an asynchronous
> socket library for Guile. I was thinking in a pure scheme library (not
> bindings for libevent or some other).
>
> Thanks,
>
> Aleix
>
>

I spent some time working on nonblocking input from a socket for my X
bindings. You can do it in pure Scheme if you're willing to live without
epoll(): Guile includes bindings for both select() and poll() (the
latter is in the undocumented (ice-9 poll) module). I used select()
because it had a section in the manual; I figure it's more "official"
and the docs make it easier for other people to follow the code.

I'm not sure how generally applicable all of my code is since it's
listening specifically for X events and replies, but the way I wait on
the socket looks like this. You have to handle interrupts manually:

(define (file-ready? fd)
    (define (do-select)
      (select (list fd) '() '()   )) 
 ;; ----------------------------^ [SECS] [USECS]
 ;; You can include an optional timeout in seconds and one in microseconds
    (define (on-error . args)
       (if (= (system-error-errno args) EINTR)
         '(() () ()) ;; This is what select returns when the file's unavailable
         (apply throw args)))
    (memq fd (car (catch 'system-error do-select on-error))))

(if (file-ready? fd)
    ... do the file operation ...
    ... loop or do something else instead ...)

Reading and writing to a socket seems to lend itself well to custom
binary ports, but then I think you're stuck with an opaque buffer -- you
can't properly ask the binary port if the next read beyond one byte will
block or not, since it might have stuff buffered or it might not. I'd
love to be wrong about that if someone happens to know how to check for
the amount remaining in a custom binary port's buffer.

Because of that, my code directly uses recv! every time it wants to read
from the socket, event if it just needs one byte. I don't think that
approach makes for very elegant nonblocking i/o, but it's been
performing well enough.

You can take a look at this for some more ideas (hopefully good ones):

https://github.com/mwitmer/guile-xcb/blob/prompts/xcb/xml/connection.scm

-- 
Mark Witmer

Re: asynchronous socket library

[Mark H Weaver]

mark@markwitmer.com writes:
> Reading and writing to a socket seems to lend itself well to custom
> binary ports,

Why do you say that?  For most purposes, Guile's native file ports are
superior, and they seem the natural choice for sockets.

> but then I think you're stuck with an opaque buffer -- you
> can't properly ask the binary port if the next read beyond one byte will
> block or not, since it might have stuff buffered or it might not. I'd
> love to be wrong about that if someone happens to know how to check for
> the amount remaining in a custom binary port's buffer.

'get-bytevector-some' from (rnrs io ports) might do what you need.  If
there's at least one byte available it will not block, but in practice
it reads much more than that (for buffered ports).  Guile 2.0.9 has a
nice fast implementation.  Prior to 2.0.9, it was less so.

In more detail: as of 2.0.9, 'get-bytevector-some' will simply copy the
input buffer managed by Guile into a bytevector and return it.  If
Guile's input buffer is empty, it will try to refill the buffer using
the port type's 'fill_input' method.  For file ports this is
fports.c:fport_fill_input, and for custom binary ports it's
r6rs-ports.c:cbip_fill_input.

Prior to 2.0.9, 'get-bytevector-some' was instead implemented as a loop
that repeatedly called 'char-ready?' and read one byte at a time.  This
would drain more than just Guile's buffer; it would try to drain the
upstream buffers as well.

However, 'char-ready?', and thus the pre-2.0.9 'byte-vector-some',
depend upon the port type providing an 'input_waiting' method.  File
ports provide this, implemented using 'poll' with 0 timeout, see
fports.c:fport_input_waiting.

Unfortunately, custom binary ports do NOT provide an 'input_waiting'
method, so 'char-ready?' always returns true, and I guess this
ultimately means that the pre-2.0.9 'get-bytevector-some' will block
until it reaches EOF.

So again I ask, why use custom binary ports for sockets?

    Regards,
      Mark

Re: asynchronous socket library

[mark]

Mark H Weaver <mhw@netris.org> writes:

> mark@markwitmer.com writes:
>> Reading and writing to a socket seems to lend itself well to custom
>> binary ports,
>
> Why do you say that?  For most purposes, Guile's native file ports are
> superior, and they seem the natural choice for sockets.
>

I over-generalized. In my application, I'm using the X protocol, which
is a binary one. I've been so involved with that I forgot that much of
the time you want to read/write character data from a socket.

> 'get-bytevector-some' from (rnrs io ports) might do what you need.  If
> there's at least one byte available it will not block, but in practice
> it reads much more than that (for buffered ports).  Guile 2.0.9 has a
> nice fast implementation.  Prior to 2.0.9, it was less so.
>
> In more detail: as of 2.0.9, 'get-bytevector-some' will simply copy the
> input buffer managed by Guile into a bytevector and return it.  If
> Guile's input buffer is empty, it will try to refill the buffer using
> the port type's 'fill_input' method.  For file ports this is
> fports.c:fport_fill_input, and for custom binary ports it's
> r6rs-ports.c:cbip_fill_input.

In order to know if the port has at least one byte ready, you need to be
able to call 'char-ready?', don't you? And as you mentioned, that
doesn't work on custom binary ports. I'm still exploring this part of
Guile so I might not have the details right.

>
> So again I ask, why use custom binary ports for sockets?
>

Thanks for the detailed explanation; I think I'll move away from
them. Were they added just to meet the r6rs spec? Are there any good
uses for custom binary ports that you're aware of?

-- 
Mark Witmer

Re: asynchronous socket library

[Mark H Weaver]

Hi Mark,

Sorry for taking so long to respond.

mark@markwitmer.com writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> mark@markwitmer.com writes:
>>> Reading and writing to a socket seems to lend itself well to custom
>>> binary ports,
>>
>> Why do you say that?  For most purposes, Guile's native file ports are
>> superior, and they seem the natural choice for sockets.
>>
>
> I over-generalized. In my application, I'm using the X protocol, which
> is a binary one. I've been so involved with that I forgot that much of
> the time you want to read/write character data from a socket.

Guile's native file ports (which also work on sockets) can be used for
binary data as well as character data.  There's no need to create a
custom binary port for this purpose.  Indeed, the custom ports have
several major disadvantages compared to the native file ports.

>> 'get-bytevector-some' from (rnrs io ports) might do what you need.  If
>> there's at least one byte available it will not block, but in practice
>> it reads much more than that (for buffered ports).  Guile 2.0.9 has a
>> nice fast implementation.  Prior to 2.0.9, it was less so.
>>
>> In more detail: as of 2.0.9, 'get-bytevector-some' will simply copy the
>> input buffer managed by Guile into a bytevector and return it.  If
>> Guile's input buffer is empty, it will try to refill the buffer using
>> the port type's 'fill_input' method.  For file ports this is
>> fports.c:fport_fill_input, and for custom binary ports it's
>> r6rs-ports.c:cbip_fill_input.
>
> In order to know if the port has at least one byte ready, you need to be
> able to call 'char-ready?', don't you? And as you mentioned, that
> doesn't work on custom binary ports. I'm still exploring this part of
> Guile so I might not have the details right.
>
>>
>> So again I ask, why use custom binary ports for sockets?
>>
>
> Thanks for the detailed explanation; I think I'll move away from
> them. Were they added just to meet the r6rs spec? Are there any good
> uses for custom binary ports that you're aware of?

Yes, custom ports are needed if the data that's being read is not coming
from a file, pipe, device, socket, string, or bytevector (the cases
covered by Guile's existing native port types; maybe I'm forgetting
some).  Similarly for writing.

For example, suppose you wanted a make an input port that reads
keypresses from a GUI interface.  In this case, the keypresses are
received via events during the main event loop.  There is no underlying
POSIX file descriptor to read from, nor is there a single Scheme string
or bytevector.  You could use pipes I suppose, but that assumes POSIX.
Another way is to make a custom port.

In the case of guile-xcb, I see two places where custom ports are used
in auth.scm.  Here's the first case:

--8<---------------cut here---------------start------------->8---
    (define auth-file (open-file (getenv "XAUTHORITY") "rb"))

    (define port
      (make-custom-binary-input-port
       "xcb-auth-input"
       (lambda (bv start count)
         (do ((n 0 (1+ n))
              (ch (read-char auth-file) (read-char auth-file)))
             ((or (>= n count) (eof-object? ch)) n)
           (bytevector-u8-set!
            bv (+ start n) (char->integer ch))))
       #f #f (lambda () (close-port port))))
--8<---------------cut here---------------end--------------->8---

First of all, that '(close-port port)' is wrong.  Here you are trying to
tell Guile how to close 'port', and this just recurses.  I guess it
should be (close-port auth-file)'.

However, I don't see why this custom port is needed at all.  You've
already opened 'auth-file' in binary mode, so you should be able to use
it directly, i.e. replace the above code with:

    (define port (open-file (getenv "XAUTHORITY") "rb"))

The other case is here (also in auth.scm):

--8<---------------cut here---------------start------------->8---
    (define (read! bv start count)
      (let* ((in-bv (make-bytevector count))
             (bytes-read (recv! sock in-bv)))
        (bytevector-copy! in-bv 0 bv start bytes-read)
        bytes-read))
    (define port (make-custom-binary-input-port "xcb-input" read! #f #f #f))
--8<---------------cut here---------------end--------------->8---

Again, I don't see why this custom port should be needed.
You should be able to use 'sock' directly.

Anyway, thanks for working on this! :)

     Regards,
       Mark

Re: asynchronous socket library

[mark.d.witmer]

Mark H Weaver <mhw@netris.org> writes:

>
> First of all, that '(close-port port)' is wrong.  Here you are trying to
> tell Guile how to close 'port', and this just recurses.  I guess it
> should be (close-port auth-file)'.

That's right. It's two bugs, actually---closing the file would cause an
infinite loop and I didn't catch it because I never actually closed the file!

>
> However, I don't see why this custom port is needed at all.  You've
> already opened 'auth-file' in binary mode, so you should be able to use
> it directly, i.e. replace the above code with:
>
>     (define port (open-file (getenv "XAUTHORITY") "rb"))
>

Ok, I finally get what you're saying... procedures like
'get-bytevector-n' or 'put-u8' and so forth work just fine on ports that
are opened in binary mode---no custom binary ports needed! 

Reading the manual gave me the impression that there was a fundamental
difference between file ports opened in binary mode as described in
section 6.14.9.1, and R6RS binary input/output ports as described in
sections 6.14.10.8 and 6.14.10.11. Now that you've explained it to me,
it makes sense that procedures that call for the latter would work on
the former as well.

-- 
Mark Witmer