Re: Re: How to notice abrupt tcp connection losses in server/client?

Re: Re: How to notice abrupt tcp connection losses in server/client?

[Zelphir Kaltstahl]

On 21.06.2018 18:00, guile-user-request@gnu.org wrote:
> Message: 5
> Date: Thu, 21 Jun 2018 16:08:20 +0100
> From: Chris Vine <vine35792468@gmail.com>
> To: guile-user@gnu.org
> Subject: Re: How to notice abrupt tcp connection losses in
> 	server/client?
> Message-ID: <20180621160820.1359196e2d30873d4a3cc1e4@gmail.com>
> Content-Type: text/plain; charset=US-ASCII
>
> On Thu, 21 Jun 2018 08:22:36 +0200
> Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
>> Hello Guile users,
>>
>> I wrote some TCP server and client in Guile which I have uploaded here:
>>
>> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-client.scm
>>
>> and here:
>>
>> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-server.scm
>>
>> or normal GitLab view:
>>
>> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/tree/dev/network-programming
>>
>> (Is it OK to post these as links, or always better to include all
>> relevant source code on the mailing list? On other e-mail lists I have
>> experienced that my message was too long and got truncated, so I posted
>> the code as links to the raw file on GitLab.)
>>
>> The loop for reacting on messages from a client on the server looks like
>> this:
>>
>> (while #t
>>   (let* ([bytes-count (recv! in-out-sock receive-buffer)]
>>          [message-received (byte-vector->utf8-message receive-buffer bytes-count)])
>>     (message-handler client-connection message-received))))))
>>
>> But this has a problem: When I run both client and server in two
>> terminals and then exit the client using Ctrl-D, the server somehow gets
>> stuck in the loop always receiving the empty string. Since that is 0
>> Bytes long, it does not really take anything from the socket (if I
>> understand correctly), but instead can recv! in the next iteration again
>> immediately. The only thing that works then for stopping this loop is to
>> hold down Ctrl-C on the server for a second or so.
>> The same happens for the client receiving message loop, because that one
>> also does not detect me suddenly interrupting or killing the server and
>> then loops on the empty string.
>> At first I thought if I caught eof-object? and then (break) the loop it
>> would solve the problem, but apparently it does not.
>> Basically I would like the server and client to be prepared for non
>> proper shutdown of either.
>>
>> How can I handle / detect abrupt connection losses, so that I can break
>> the message handling loop?
> The POSIX recv() function returns 0 on end of file (connection closed)
> so I expect the scheme recv! procedure does the same.  So on end-of-file
> your code appears to be producing an endless supply of empty strings.
I actually tried to (break) the loop when the received message is
(eof-object? ...), but it might be that I used this predicate at the
wrong place (directly inside the loop when the message was already
copied from the receive-buffer, instead of when recv!, I believe), now
that I think about it … and maybe that is why (break)ing the loop did
not work. Need to investigate more.
> More generally, you will find it easier to use something like the
> get-line procedure to read text, or something like get-bytevector-n or
> get-bytevector-n! for reading binary records.  recv! only really becomes
> important when the flags argument is meaningful.
Ah ok, thanks for the advice! I thought recv! was the way to go, as it
worked so nicely so far.
> Rather than starting a new thread for each connection, you will get much
> better scalability if you use something like fibers
> ( https://github.com/wingo/fibers/ ), guile-a-sync2
> ( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
> ( https://www.gnu.org/software/8sync/ ).
Yep =) I am aware. I already wrote a comment inside the code noting that
this new thread thing is probably not so good. Not sure I should
introduce fibers into this example code yet though. And looking at
fibers and 8sync is on my to-do list ; )
> Chris
I will try these things and see how it goes. Thanks!

~ Zelphir

Re: How to notice abrupt tcp connection losses in server/client?

[Chris Vine]

On Fri, 22 Jun 2018 00:19:49 +0200
Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
> On 21.06.2018 18:00, guile-user-request@gnu.org wrote:
> > From: Chris Vine <vine35792468@gmail.com>
[snip]
> > The POSIX recv() function returns 0 on end of file (connection closed)
> > so I expect the scheme recv! procedure does the same.  So on end-of-file
> > your code appears to be producing an endless supply of empty strings.
>
> I actually tried to (break) the loop when the received message is
> (eof-object? ...), but it might be that I used this predicate at the
> wrong place (directly inside the loop when the message was already
> copied from the receive-buffer, instead of when recv!, I believe), now
> that I think about it … and maybe that is why (break)ing the loop did
> not work. Need to investigate more.

It looks as if recv! returns 0 on end-of-file, not an eof-object, so
that may be your problem.  That is not the case with get-line or
get-bytevector-n.  Note that recv! may also return less than the
requested number of bytes - it does not guarantee to block until the
entire request has been met or end-of-file is reached, as it is just a
wrapper for POSIX/windows recv().

I think you will need to use recv! with windows, because you cannot
read from sockets in windows using POSIX read().  With other OS's it
would be more normal to use the R5RS/R6RS port procedures, because they
are easier to use and are buffered.

For asynchronous non-blocking ports, I am not sure whether recv! is safe
with guile-2.2's suspendable ports or not.  The get-line and
get-bytevector-n procedures are.  The get-bytevector-n! procedure is
not.  I suspect recv! may not be also.

> > More generally, you will find it easier to use something like the
> > get-line procedure to read text, or something like get-bytevector-n or
> > get-bytevector-n! for reading binary records.  recv! only really becomes
> > important when the flags argument is meaningful.

> Ah ok, thanks for the advice! I thought recv! was the way to go, as it
> worked so nicely so far.

A call to unix read() on a socket is equivalent to a call to recv()
with default flags.  At the end of it, the buffered guile port reading
procedures generally involve a call to read().

> > Rather than starting a new thread for each connection, you will get much
> > better scalability if you use something like fibers
> > ( https://github.com/wingo/fibers/ ), guile-a-sync2
> > ( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
> > ( https://www.gnu.org/software/8sync/ ).
> Yep =) I am aware. I already wrote a comment inside the code noting that
> this new thread thing is probably not so good. Not sure I should
> introduce fibers into this example code yet though. And looking at
> fibers and 8sync is on my to-do list ; )
> > Chris
> I will try these things and see how it goes. Thanks!

The guile-a-sync/guile-a-sync2 libraries (of which I am the author) have
an example of a client and server in the doc directory, which might be
of interest to you.  fibers is definitely worth a look also, but is
linux only, and also guile-2.2 only.  I think it also has some examples
(not sure).

Chris

Re: How to notice abrupt tcp connection losses in server/client?

[Eli Zaretskii]

> Date: Fri, 22 Jun 2018 00:09:16 +0100
> From: Chris Vine <vine35792468@gmail.com>
> 
> I think you will need to use recv! with windows, because you cannot
> read from sockets in windows using POSIX read().

What makes you say that?  It isn't true; Gawk does that on Windows,
and it works very well.  The only two issues, which are easy to
overcome with wrappers or macros, are:

  . you need to create the socket with 'WSASocket; rather than
    'socket', since the latter creates overlapped sockets that cannot
    be used with file I/O APIs;
  . you need to convert the SOCKET type (which is a handle in
    disguise) into a file descriptor and back using a pair of library
    functions

Perhaps Guile doesn't yet do that (I didn't look, it could be in
Gnulib functions), but it would be easy to add if so.

Re: How to notice abrupt tcp connection losses in server/client?

[Chris Vine]

On Fri, 22 Jun 2018 09:27:30 +0300
Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Fri, 22 Jun 2018 00:09:16 +0100
> > From: Chris Vine <vine35792468@gmail.com>
> > 
> > I think you will need to use recv! with windows, because you cannot
> > read from sockets in windows using POSIX read().
> 
> What makes you say that?  It isn't true; Gawk does that on Windows,
> and it works very well.  The only two issues, which are easy to
> overcome with wrappers or macros, are:
> 
>   . you need to create the socket with 'WSASocket; rather than
>     'socket', since the latter creates overlapped sockets that cannot
>     be used with file I/O APIs;
>   . you need to convert the SOCKET type (which is a handle in
>     disguise) into a file descriptor and back using a pair of library
>     functions
> 
> Perhaps Guile doesn't yet do that (I didn't look, it could be in
> Gnulib functions), but it would be easy to add if so.

OK, my misunderstanding then.  Looking at the wrappers for windows
sockets (lib/socket.c) it looks as if it may work - since you use
windows and I don't, why not try it and tell us?

Re: Re: How to notice abrupt tcp connection losses in server/client?

[Zelphir Kaltstahl]

On 22.06.2018 18:00, guile-user-request@gnu.org wrote:
> Message: 2
> Date: Fri, 22 Jun 2018 00:09:16 +0100
> From: Chris Vine <vine35792468@gmail.com>
> To: guile-user@gnu.org
> Subject: Re: How to notice abrupt tcp connection losses in
> 	server/client?
> Message-ID: <20180622000916.e5180d431446d15843d3f1e8@gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Fri, 22 Jun 2018 00:19:49 +0200
> Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
>> On 21.06.2018 18:00, guile-user-request@gnu.org wrote:
>>> From: Chris Vine <vine35792468@gmail.com>
> [snip]
>>> The POSIX recv() function returns 0 on end of file (connection closed)
>>> so I expect the scheme recv! procedure does the same.  So on end-of-file
>>> your code appears to be producing an endless supply of empty strings.
>> I actually tried to (break) the loop when the received message is
>> (eof-object? ...), but it might be that I used this predicate at the
>> wrong place (directly inside the loop when the message was already
>> copied from the receive-buffer, instead of when recv!, I believe), now
>> that I think about it ? and maybe that is why (break)ing the loop did
>> not work. Need to investigate more.
> It looks as if recv! returns 0 on end-of-file, not an eof-object, so
> that may be your problem.  That is not the case with get-line or
> get-bytevector-n.  Note that recv! may also return less than the
> requested number of bytes - it does not guarantee to block until the
> entire request has been met or end-of-file is reached, as it is just a
> wrapper for POSIX/windows recv().
I think you are exactly right. If it returns 0 for end of file (what is
the difference between this and end of file object?), then 0 bytes will
be copied to the receive byte vector. When converting that to utf8
string, it will be the empty string and that is printed. However, since
there is still no new message (client connection is no more) there is
still only EOF and the same will happen over and over again.

> I think you will need to use recv! with windows, because you cannot
> read from sockets in windows using POSIX read().  With other OS's it
> would be more normal to use the R5RS/R6RS port procedures, because they
> are easier to use and are buffered.
>
> For asynchronous non-blocking ports, I am not sure whether recv! is safe
> with guile-2.2's suspendable ports or not.  The get-line and
> get-bytevector-n procedures are.  The get-bytevector-n! procedure is
> not.  I suspect recv! may not be also.
>
>>> More generally, you will find it easier to use something like the
>>> get-line procedure to read text, or something like get-bytevector-n or
>>> get-bytevector-n! for reading binary records.  recv! only really becomes
>>> important when the flags argument is meaningful.
>> Ah ok, thanks for the advice! I thought recv! was the way to go, as it
>> worked so nicely so far.
> A call to unix read() on a socket is equivalent to a call to recv()
> with default flags.  At the end of it, the buffered guile port reading
> procedures generally involve a call to read().
>
>>> Rather than starting a new thread for each connection, you will get much
>>> better scalability if you use something like fibers
>>> ( https://github.com/wingo/fibers/ ), guile-a-sync2
>>> ( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
>>> ( https://www.gnu.org/software/8sync/ ).
>> Yep =) I am aware. I already wrote a comment inside the code noting that
>> this new thread thing is probably not so good. Not sure I should
>> introduce fibers into this example code yet though. And looking at
>> fibers and 8sync is on my to-do list ; )
>>> Chris
>> I will try these things and see how it goes. Thanks!
> The guile-a-sync/guile-a-sync2 libraries (of which I am the author) have
> an example of a client and server in the doc directory, which might be
> of interest to you.  fibers is definitely worth a look also, but is
> linux only, and also guile-2.2 only.  I think it also has some examples
> (not sure).
>
> Chris

I now use get-bytevector-n with the count which was the size of the
receive-buffer and it does not have the endless looping behavior. All
seems fine now. Thanks!

Re: How to notice abrupt tcp connection losses in server/client?

[Chris Vine]

On Fri, 22 Jun 2018 22:17:51 +0200
Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
[snip]
> ... it returns 0 for end of file (what is the difference between this
> and end of file object?

R6RS states that "The end-of-file object is returned by various I/O
procedures when they reach end of file".  It can be obtained by invoking
the R6RS eof-object procedure and can be queried with the eof-object?
predicate, so '(eof-object? (eof-object))' will evaluate to true.  R6RS
also says "The end-of-file object is not a datum value, and thus has no
external representation".

0 is an integer.  POSIX read() and recv() indicate end-of-file by
unblocking and returning 0.  Subsequent calls to read() or recv() might
return -1 indicating an error, in which case recv! would throw an
exception.

How to notice abrupt tcp connection losses in server/client?

[Zelphir Kaltstahl]

Hello Guile users,

I wrote some TCP server and client in Guile which I have uploaded here:

https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-client.scm

and here:

https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-server.scm

or normal GitLab view:

https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/tree/dev/network-programming

(Is it OK to post these as links, or always better to include all
relevant source code on the mailing list? On other e-mail lists I have
experienced that my message was too long and got truncated, so I posted
the code as links to the raw file on GitLab.)

The loop for reacting on messages from a client on the server looks like
this:

(while #t
  (let* ([bytes-count (recv! in-out-sock receive-buffer)]
         [message-received (byte-vector->utf8-message receive-buffer bytes-count)])
    (message-handler client-connection message-received))))))

But this has a problem: When I run both client and server in two
terminals and then exit the client using Ctrl-D, the server somehow gets
stuck in the loop always receiving the empty string. Since that is 0
Bytes long, it does not really take anything from the socket (if I
understand correctly), but instead can recv! in the next iteration again
immediately. The only thing that works then for stopping this loop is to
hold down Ctrl-C on the server for a second or so.
The same happens for the client receiving message loop, because that one
also does not detect me suddenly interrupting or killing the server and
then loops on the empty string.
At first I thought if I caught eof-object? and then (break) the loop it
would solve the problem, but apparently it does not.
Basically I would like the server and client to be prepared for non
proper shutdown of either.

How can I handle / detect abrupt connection losses, so that I can break
the message handling loop?

On 20.06.2018 18:00, guile-user-request@gnu.org wrote:
> Send guile-user mailing list submissions to
> 	guile-user@gnu.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.gnu.org/mailman/listinfo/guile-user
> or, via email, send a message with subject or body 'help' to
> 	guile-user-request@gnu.org
>
> You can reach the person managing the list at
> 	guile-user-owner@gnu.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of guile-user digest..."
>
>
> Today's Topics:
>
>    1. Re: lat? and atom? not in guile? (Thompson, David)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 20 Jun 2018 10:55:43 -0400
> From: "Thompson, David" <dthompson2@worcester.edu>
> To: John Cowan <cowan@ccil.org>
> Cc: Joshua Branson <jbranso@fastmail.com>, Guile User
> 	<guile-user@gnu.org>
> Subject: Re: lat? and atom? not in guile?
> Message-ID:
> 	<CAJ=RwfZxx_ihGPVmij+NV0G6JrQ2Z-SOzfjGObotQZS0q_JG1w@mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
>
> On Wed, Jun 20, 2018 at 10:40 AM, John Cowan <cowan@ccil.org> wrote:
>> On Wed, Jun 20, 2018 at 10:37 AM, Joshua Branson <jbranso@fastmail.com>
>> wrote:
>>
>>
>>> Oh hey David!  Thanks for the explanation!  Maybe I need to take another
>>> look at that book then.  I found it a little repetitive, but I am
>>> probably not quite grasping some of the fundamentals.
>>>
>> It's meant to be repetitive.  It's explicitly compared in the introduction
>> to Hanon's finger exercises for the piano:  C-E-F-G-A-G-F-E,
>> D-F-G-A-B-A-G-F, E-G-A-B-C-B-A-D and so on forever, up the scale and down
>> in every key.  Boring as hell, but just the thing to get fluency into your
>> fingers.
> It's repetitive but honestly I found that the comedic writing made the
> exercises very entertaining. I was rarely bored.  I don't play the
> piano, but I do play drums, and the equivalent to Hanon's finger
> exercises is an old book called Stick Control, which is definitely in
> the boring as hell category! I'm glad that Lisp beginners have
> something more fun available. :)
>
> - Dave
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> guile-user mailing list
> guile-user@gnu.org
> https://lists.gnu.org/mailman/listinfo/guile-user
>
>
> ------------------------------
>
> End of guile-user Digest, Vol 187, Issue 14
> *******************************************

Re: How to notice abrupt tcp connection losses in server/client?

[Joshua Branson]

Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> writes:

> Hello Guile users,
>
> I wrote some TCP server and client in Guile which I have uploaded here:
>
> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-client.scm
>
> and here:
>
> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-server.scm
>
> or normal GitLab view:
>
> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/tree/dev/network-programming
>

I'll have to check these out!  Thanks for mentioning them!

Re: How to notice abrupt tcp connection losses in server/client?

[Chris Vine]

On Thu, 21 Jun 2018 08:22:36 +0200
Zelphir Kaltstahl <zelphirkaltstahl@gmail.com> wrote:
> Hello Guile users,
> 
> I wrote some TCP server and client in Guile which I have uploaded here:
> 
> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-client.scm
> 
> and here:
> 
> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/raw/dev/network-programming/tcp-server.scm
> 
> or normal GitLab view:
> 
> https://gitlab.com/zelphir-kaltstahl-projects/guile-scheme-tutorials-and-examples/tree/dev/network-programming
> 
> (Is it OK to post these as links, or always better to include all
> relevant source code on the mailing list? On other e-mail lists I have
> experienced that my message was too long and got truncated, so I posted
> the code as links to the raw file on GitLab.)
> 
> The loop for reacting on messages from a client on the server looks like
> this:
> 
> (while #t
>   (let* ([bytes-count (recv! in-out-sock receive-buffer)]
>          [message-received (byte-vector->utf8-message receive-buffer bytes-count)])
>     (message-handler client-connection message-received))))))
> 
> But this has a problem: When I run both client and server in two
> terminals and then exit the client using Ctrl-D, the server somehow gets
> stuck in the loop always receiving the empty string. Since that is 0
> Bytes long, it does not really take anything from the socket (if I
> understand correctly), but instead can recv! in the next iteration again
> immediately. The only thing that works then for stopping this loop is to
> hold down Ctrl-C on the server for a second or so.
> The same happens for the client receiving message loop, because that one
> also does not detect me suddenly interrupting or killing the server and
> then loops on the empty string.
> At first I thought if I caught eof-object? and then (break) the loop it
> would solve the problem, but apparently it does not.
> Basically I would like the server and client to be prepared for non
> proper shutdown of either.
> 
> How can I handle / detect abrupt connection losses, so that I can break
> the message handling loop?

The POSIX recv() function returns 0 on end of file (connection closed)
so I expect the scheme recv! procedure does the same.  So on end-of-file
your code appears to be producing an endless supply of empty strings.

More generally, you will find it easier to use something like the
get-line procedure to read text, or something like get-bytevector-n or
get-bytevector-n! for reading binary records.  recv! only really becomes
important when the flags argument is meaningful.

Rather than starting a new thread for each connection, you will get much
better scalability if you use something like fibers
( https://github.com/wingo/fibers/ ), guile-a-sync2
( https://github.com/ChrisVine/guile-a-sync2/ ) or 8sync
( https://www.gnu.org/software/8sync/ ).

Chris