Re: Re: How to notice abrupt tcp connection losses in server/client?

Re: Re: How to notice abrupt tcp connection losses in server/client?

[Zelphir Kaltstahl]

On 21.06.2018 18:00, guile-user-request@gnu.org wrote:
> Message: 5
> Date: Thu, 21 Jun 2018 16:08:20 +0100
> From: Chris Vine <vine35792468@gmail.com>
> To: guile-user@gnu.org
> Subject: Re: How to notice abrupt tcp connection losses in
> 	server/client?
> Message-ID: <20180621160820.1359196e2d30873d4a3cc1e4@gmail.com>
> Content-Type: text/plain; charset=US-ASCII
>
> On Thu, 21 Jun 2018 08:22:36 +0200
> Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
>> Hello Guile users,
>>
>> I wrote some TCP server and client in Guile which I have uploaded here:
>>
>> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-client.scm
>>
>> and here:
>>
>> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-server.scm
>>
>> or normal GitLab view:
>>
>> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/tree/dev/network-programming
>>
>> (Is it OK to post these as links, or always better to include all
>> relevant source code on the mailing list? On other e-mail lists I have
>> experienced that my message was too long and got truncated, so I posted
>> the code as links to the raw file on GitLab.)
>>
>> The loop for reacting on messages from a client on the server looks like
>> this:
>>
>> (while #t
>>   (let* ([bytes-count (recv! in-out-sock receive-buffer)]
>>          [message-received (byte-vector->utf8-message receive-buffer bytes-count)])
>>     (message-handler client-connection message-received))))))
>>
>> But this has a problem: When I run both client and server in two
>> terminals and then exit the client using Ctrl-D, the server somehow gets
>> stuck in the loop always receiving the empty string. Since that is 0
>> Bytes long, it does not really take anything from the socket (if I
>> understand correctly), but instead can recv! in the next iteration again
>> immediately. The only thing that works then for stopping this loop is to
>> hold down Ctrl-C on the server for a second or so.
>> The same happens for the client receiving message loop, because that one
>> also does not detect me suddenly interrupting or killing the server and
>> then loops on the empty string.
>> At first I thought if I caught eof-object? and then (break) the loop it
>> would solve the problem, but apparently it does not.
>> Basically I would like the server and client to be prepared for non
>> proper shutdown of either.
>>
>> How can I handle / detect abrupt connection losses, so that I can break
>> the message handling loop?
> The POSIX recv() function returns 0 on end of file (connection closed)
> so I expect the scheme recv! procedure does the same.  So on end-of-file
> your code appears to be producing an endless supply of empty strings.
I actually tried to (break) the loop when the received message is
(eof-object? ...), but it might be that I used this predicate at the
wrong place (directly inside the loop when the message was already
copied from the receive-buffer, instead of when recv!, I believe), now
that I think about it … and maybe that is why (break)ing the loop did
not work. Need to investigate more.
> More generally, you will find it easier to use something like the
> get-line procedure to read text, or something like get-bytevector-n or
> get-bytevector-n! for reading binary records.  recv! only really becomes
> important when the flags argument is meaningful.
Ah ok, thanks for the advice! I thought recv! was the way to go, as it
worked so nicely so far.
> Rather than starting a new thread for each connection, you will get much
> better scalability if you use something like fibers
> ( https://github.com/wingo/fibers/ ), guile-a-sync2
> ( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
> ( https://www.gnu.org/software/8sync/ ).
Yep =) I am aware. I already wrote a comment inside the code noting that
this new thread thing is probably not so good. Not sure I should
introduce fibers into this example code yet though. And looking at
fibers and 8sync is on my to-do list ; )
> Chris
I will try these things and see how it goes. Thanks!

~ Zelphir

Re: How to notice abrupt tcp connection losses in server/client?

[Chris Vine]

On Fri, 22 Jun 2018 00:19:49 +0200
Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
> On 21.06.2018 18:00, guile-user-request@gnu.org wrote:
> > From: Chris Vine <vine35792468@gmail.com>
[snip]
> > The POSIX recv() function returns 0 on end of file (connection closed)
> > so I expect the scheme recv! procedure does the same.  So on end-of-file
> > your code appears to be producing an endless supply of empty strings.
>
> I actually tried to (break) the loop when the received message is
> (eof-object? ...), but it might be that I used this predicate at the
> wrong place (directly inside the loop when the message was already
> copied from the receive-buffer, instead of when recv!, I believe), now
> that I think about it … and maybe that is why (break)ing the loop did
> not work. Need to investigate more.

It looks as if recv! returns 0 on end-of-file, not an eof-object, so
that may be your problem.  That is not the case with get-line or
get-bytevector-n.  Note that recv! may also return less than the
requested number of bytes - it does not guarantee to block until the
entire request has been met or end-of-file is reached, as it is just a
wrapper for POSIX/windows recv().

I think you will need to use recv! with windows, because you cannot
read from sockets in windows using POSIX read().  With other OS's it
would be more normal to use the R5RS/R6RS port procedures, because they
are easier to use and are buffered.

For asynchronous non-blocking ports, I am not sure whether recv! is safe
with guile-2.2's suspendable ports or not.  The get-line and
get-bytevector-n procedures are.  The get-bytevector-n! procedure is
not.  I suspect recv! may not be also.

> > More generally, you will find it easier to use something like the
> > get-line procedure to read text, or something like get-bytevector-n or
> > get-bytevector-n! for reading binary records.  recv! only really becomes
> > important when the flags argument is meaningful.

> Ah ok, thanks for the advice! I thought recv! was the way to go, as it
> worked so nicely so far.

A call to unix read() on a socket is equivalent to a call to recv()
with default flags.  At the end of it, the buffered guile port reading
procedures generally involve a call to read().

> > Rather than starting a new thread for each connection, you will get much
> > better scalability if you use something like fibers
> > ( https://github.com/wingo/fibers/ ), guile-a-sync2
> > ( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
> > ( https://www.gnu.org/software/8sync/ ).
> Yep =) I am aware. I already wrote a comment inside the code noting that
> this new thread thing is probably not so good. Not sure I should
> introduce fibers into this example code yet though. And looking at
> fibers and 8sync is on my to-do list ; )
> > Chris
> I will try these things and see how it goes. Thanks!

The guile-a-sync/guile-a-sync2 libraries (of which I am the author) have
an example of a client and server in the doc directory, which might be
of interest to you.  fibers is definitely worth a look also, but is
linux only, and also guile-2.2 only.  I think it also has some examples
(not sure).

Chris

Re: How to notice abrupt tcp connection losses in server/client?

[Eli Zaretskii]

> Date: Fri, 22 Jun 2018 00:09:16 +0100
> From: Chris Vine <vine35792468@gmail.com>
> 
> I think you will need to use recv! with windows, because you cannot
> read from sockets in windows using POSIX read().

What makes you say that?  It isn't true; Gawk does that on Windows,
and it works very well.  The only two issues, which are easy to
overcome with wrappers or macros, are:

  . you need to create the socket with 'WSASocket; rather than
    'socket', since the latter creates overlapped sockets that cannot
    be used with file I/O APIs;
  . you need to convert the SOCKET type (which is a handle in
    disguise) into a file descriptor and back using a pair of library
    functions

Perhaps Guile doesn't yet do that (I didn't look, it could be in
Gnulib functions), but it would be easy to add if so.

Re: How to notice abrupt tcp connection losses in server/client?

[Chris Vine]

On Fri, 22 Jun 2018 09:27:30 +0300
Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Fri, 22 Jun 2018 00:09:16 +0100
> > From: Chris Vine <vine35792468@gmail.com>
> > 
> > I think you will need to use recv! with windows, because you cannot
> > read from sockets in windows using POSIX read().
> 
> What makes you say that?  It isn't true; Gawk does that on Windows,
> and it works very well.  The only two issues, which are easy to
> overcome with wrappers or macros, are:
> 
>   . you need to create the socket with 'WSASocket; rather than
>     'socket', since the latter creates overlapped sockets that cannot
>     be used with file I/O APIs;
>   . you need to convert the SOCKET type (which is a handle in
>     disguise) into a file descriptor and back using a pair of library
>     functions
> 
> Perhaps Guile doesn't yet do that (I didn't look, it could be in
> Gnulib functions), but it would be easy to add if so.

OK, my misunderstanding then.  Looking at the wrappers for windows
sockets (lib/socket.c) it looks as if it may work - since you use
windows and I don't, why not try it and tell us?

Re: Re: How to notice abrupt tcp connection losses in server/client?

[Zelphir Kaltstahl]

On 22.06.2018 18:00, guile-user-request@gnu.org wrote:
> Message: 2
> Date: Fri, 22 Jun 2018 00:09:16 +0100
> From: Chris Vine <vine35792468@gmail.com>
> To: guile-user@gnu.org
> Subject: Re: How to notice abrupt tcp connection losses in
> 	server/client?
> Message-ID: <20180622000916.e5180d431446d15843d3f1e8@gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Fri, 22 Jun 2018 00:19:49 +0200
> Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
>> On 21.06.2018 18:00, guile-user-request@gnu.org wrote:
>>> From: Chris Vine <vine35792468@gmail.com>
> [snip]
>>> The POSIX recv() function returns 0 on end of file (connection closed)
>>> so I expect the scheme recv! procedure does the same.  So on end-of-file
>>> your code appears to be producing an endless supply of empty strings.
>> I actually tried to (break) the loop when the received message is
>> (eof-object? ...), but it might be that I used this predicate at the
>> wrong place (directly inside the loop when the message was already
>> copied from the receive-buffer, instead of when recv!, I believe), now
>> that I think about it ? and maybe that is why (break)ing the loop did
>> not work. Need to investigate more.
> It looks as if recv! returns 0 on end-of-file, not an eof-object, so
> that may be your problem.  That is not the case with get-line or
> get-bytevector-n.  Note that recv! may also return less than the
> requested number of bytes - it does not guarantee to block until the
> entire request has been met or end-of-file is reached, as it is just a
> wrapper for POSIX/windows recv().
I think you are exactly right. If it returns 0 for end of file (what is
the difference between this and end of file object?), then 0 bytes will
be copied to the receive byte vector. When converting that to utf8
string, it will be the empty string and that is printed. However, since
there is still no new message (client connection is no more) there is
still only EOF and the same will happen over and over again.

> I think you will need to use recv! with windows, because you cannot
> read from sockets in windows using POSIX read().  With other OS's it
> would be more normal to use the R5RS/R6RS port procedures, because they
> are easier to use and are buffered.
>
> For asynchronous non-blocking ports, I am not sure whether recv! is safe
> with guile-2.2's suspendable ports or not.  The get-line and
> get-bytevector-n procedures are.  The get-bytevector-n! procedure is
> not.  I suspect recv! may not be also.
>
>>> More generally, you will find it easier to use something like the
>>> get-line procedure to read text, or something like get-bytevector-n or
>>> get-bytevector-n! for reading binary records.  recv! only really becomes
>>> important when the flags argument is meaningful.
>> Ah ok, thanks for the advice! I thought recv! was the way to go, as it
>> worked so nicely so far.
> A call to unix read() on a socket is equivalent to a call to recv()
> with default flags.  At the end of it, the buffered guile port reading
> procedures generally involve a call to read().
>
>>> Rather than starting a new thread for each connection, you will get much
>>> better scalability if you use something like fibers
>>> ( https://github.com/wingo/fibers/ ), guile-a-sync2
>>> ( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
>>> ( https://www.gnu.org/software/8sync/ ).
>> Yep =) I am aware. I already wrote a comment inside the code noting that
>> this new thread thing is probably not so good. Not sure I should
>> introduce fibers into this example code yet though. And looking at
>> fibers and 8sync is on my to-do list ; )
>>> Chris
>> I will try these things and see how it goes. Thanks!
> The guile-a-sync/guile-a-sync2 libraries (of which I am the author) have
> an example of a client and server in the doc directory, which might be
> of interest to you.  fibers is definitely worth a look also, but is
> linux only, and also guile-2.2 only.  I think it also has some examples
> (not sure).
>
> Chris

I now use get-bytevector-n with the count which was the size of the
receive-buffer and it does not have the endless looping behavior. All
seems fine now. Thanks!