Determining whether a TCP connection is up

Determining whether a TCP connection is up

[Lars Magne Ingebrigtsen]

TCP connections may go AWOL in many different ways.  The most common
reasons are 1) changing IP address on a laptop, 2) the "router" changes
IP address when you're on a laptop tethered to a telephone, and 3) the
server dies.

In some cases (like in 1 and 2) you will perhaps get a TCP reset from
the server when you issue a command.  That's fine, and solves the
problem.

But, these days, most services are behind firewalls that just drop
packages that come on connections it doesn't know about.  So you get no
packets in return.  The same is true for 3).

So the only way to see whether a TCP connection is up is really to send
a command and see whether you get anything back.

Having to implement that kind of logic for all the network protocols is
tedious and can't be done generally.

For one, you have to establish reasonable timeouts, but there are none
that can be used generally.  For instance, when you send a DELETE
command to an IMAP server, it may take 20 seconds for it to remove the
message and re-create a big folder.  You don't want to wait 20 seconds
in general before saying "ok, we have a timeout and should reconnect".

So it would be fantastic if it was possible to do...  something...  to
make the following pseudo-code work:

(while (not done)
  (accept-process-output process 0 1)
  (when (and (absolutely-nothing-was-received)
             (> timeout two-seconds)
             (not (probe-for-connection-up)))
    (kill-process-and-reconnect process)
    (replay-commands)))

You get the drift.

So the problem is `probe-for-connection-up'.  Is there a general TCP way
to probe for a connection?  Can we (for instance) send a TCP KEEPALIVE
once and see whether we get anything back within (say) a second?

Or do something else that's clever?

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php

Re: Determining whether a TCP connection is up

[Paul Eggert]

On 08/03/2013 05:30 AM, Lars Magne Ingebrigtsen wrote:
> Is there a general TCP way
> to probe for a connection?  Can we (for instance) send a TCP KEEPALIVE
> once and see whether we get anything back within (say) a second?

I'm afraid not, since TCP keepalives won't work through proxies.
See, for example, the first FAQ in
http://www.proftpd.org/docs/howto/KeepAlives.html

Re: Determining whether a TCP connection is up

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Sat, Aug 03 2013, Lars Magne Ingebrigtsen wrote:

> For one, you have to establish reasonable timeouts, but there are none
> that can be used generally.  For instance, when you send a DELETE
> command to an IMAP server, it may take 20 seconds for it to remove the
> message and re-create a big folder.  You don't want to wait 20 seconds
> in general before saying "ok, we have a timeout and should reconnect".

In such a case, the other problem is that the connection is going to
block Emacs anyway, even for a good reason.

Thinking out loud:

I think most IMAP clients uses a pool of several connections in the
background to deliver actions based on some sort of queue, like delete
something or fetch this. Having this would solve partially the problem
since if a connection goes AWOL, you just have to wait for the kernel to
kill the TCP connection it at some point (which might takes time), but
you can continue to create new (working) connections to fulfil the user
requests in the meantime, and then requeue the action that timed-out.

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Determining whether a TCP connection is up

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> I think most IMAP clients uses a pool of several connections in the
> background to deliver actions based on some sort of queue, like delete
> something or fetch this. Having this would solve partially the problem
> since if a connection goes AWOL, you just have to wait for the kernel to
> kill the TCP connection it at some point (which might takes time), but
> you can continue to create new (working) connections to fulfil the user
> requests in the meantime, and then requeue the action that timed-out.

Whether you have a pool of connections or kill/reconnect, the problem
remains the same: Figuring out when to switch connections/reconnect.

And in the case that the IP address changed, all your connections in the
pool will hang, so it doesn't really help...

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php
  and http://lars.ingebrigtsen.no/2013/08/twenty-years-of-september.html

Re: Determining whether a TCP connection is up

[chad]

On 04 Aug 2013, at 19:12, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> 
> And in the case that the IP address changed, all your connections in the
> pool will hang, so it doesn't really help...

Is there a useful DBus or NetworkManager interface that will help
in the GNU/Linux case?

~Chad

Re: Determining whether a TCP connection is up

[Lars Magne Ingebrigtsen]

chad <yandros@MIT.EDU> writes:

> Is there a useful DBus or NetworkManager interface that will help
> in the GNU/Linux case?

Yes.  But it doesn't help in the "tethered" case, where your "router"
(i.e., your cell phone) changes IP, but your laptop doesn't.

That's no reason not do write code to terminate all current TCP
connections if DBus says that we've changed IP (we should do that, too),
but it's not a complete solution.

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php
  and http://lars.ingebrigtsen.no/2013/08/twenty-years-of-september.html

Re: Determining whether a TCP connection is up

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

On Mon, Aug 05 2013, Lars Magne Ingebrigtsen wrote:

> Whether you have a pool of connections or kill/reconnect, the problem
> remains the same: Figuring out when to switch connections/reconnect.
>
> And in the case that the IP address changed, all your connections in the
> pool will hang, so it doesn't really help...

It would help a lot:
- connections would be stuck in the *background*, avoiding getting your
  Emacs frozen;
- connections will eventually time out (the kernel does have a TCP
  timeout), and you will be able to reschedule the operation since you
  used a queue.

Also, sqending regular NOOP command should triggers early detection of
IP changes. If I'm not mistaken, if the kernel thinks you have a TCP
session established, but your NATed public IP address changed, the
remote server is going to send you a RST or something like that as soon
as it received something, and the kernel will consider the connection as
no more valid and will close it.

That could even work for long operation:
- send a COMMAND that takes a lot of time
- change your public network IP address
- send a NOOP (seems you can do that, check
  http://tools.ietf.org/html/rfc3501#section-5.5)
  -> if the remote server doesn't know your public IP address, the
     connection will be reset; restart!
  -> if the connection's good, the server might replies now, or later
     (see RFC)

My 2c,

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Determining whether a TCP connection is up

[Stefan Monnier]

> - connections will eventually time out (the kernel does have a TCP
>   timeout), and you will be able to reschedule the operation since you
>   used a queue.

Which reminds me that I regularly have Gnus sessions where I get
neither an answer nor a timeout after (much) more than 10 minutes.


        Stefan

Re: Determining whether a TCP connection is up

[Thierry Volpiatto]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> - connections will eventually time out (the kernel does have a TCP
>>   timeout), and you will be able to reschedule the operation since you
>>   used a queue.
>
> Which reminds me that I regularly have Gnus sessions where I get
> neither an answer nor a timeout after (much) more than 10 minutes.
It seems that killing the processes (nnimap, nntpd) each time a group is
quitted avoid a lot of C-g'ing, especially with tethering:

(defun tv-gnus-kill-all-nnimap-procs ()
  (loop for proc in (process-list)
        when (string-match "\\*?nnimap\\|nntpd" (process-name proc))
        do (delete-process proc)))
(add-hook 'gnus-exit-group-hook 'tv-gnus-kill-all-nnimap-procs)
(add-hook 'gnus-group-catchup-group-hook 'tv-gnus-kill-all-nnimap-procs)

Not perfect but it helps.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Determining whether a TCP connection is up

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

On Mon, Aug 05 2013, Stefan Monnier wrote:

>> - connections will eventually time out (the kernel does have a TCP
>>   timeout), and you will be able to reschedule the operation since you
>>   used a queue.
>
> Which reminds me that I regularly have Gnus sessions where I get
> neither an answer nor a timeout after (much) more than 10 minutes.

Well, the default TCP time-out for Linux is 2 hours. Wait a bit more. :-)

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Determining whether a TCP connection is up

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> It would help a lot:
> - connections would be stuck in the *background*, avoiding getting your
>   Emacs frozen;

Not really.  Network chatter is issued when users want something to
happen, like entering a group and the like. 

> - connections will eventually time out (the kernel does have a TCP
>   timeout), and you will be able to reschedule the operation since you
>   used a queue.

Heh heh.  After waiting half an hour?

> Also, sqending regular NOOP command should triggers early detection of
> IP changes. If I'm not mistaken, if the kernel thinks you have a TCP
> session established, but your NATed public IP address changed, the
> remote server is going to send you a RST or something like that as soon
> as it received something, and the kernel will consider the connection as
> no more valid and will close it.

Nope.  This no longer happens in modern network setups.  Like my
original article said, it's common for people to run firewalls that drop
packages that don't belong to TCP sessions.  So you get no RST.

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php
  and http://lars.ingebrigtsen.no/2013/08/twenty-years-of-september.html

Re: Determining whether a TCP connection is up

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

Did someone take a look at how other IMAP clients handles such
situation?

Weirdly enough, Gnus is the only one to show such issues, whereas I
can't recall ever having this problem in mutt nor seeing it in Evolution
or Thunderbird. And I don't believe in magic.

-- 
Julien Danjou
/* Free Software hacker * freelance consultant
   http://julien.danjou.info */

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Determining whether a TCP connection is up

[joakim]

Julien Danjou <julien@danjou.info> writes:

> Did someone take a look at how other IMAP clients handles such
> situation?
>
> Weirdly enough, Gnus is the only one to show such issues, whereas I
> can't recall ever having this problem in mutt nor seeing it in Evolution
> or Thunderbird. And I don't believe in magic.

I used to have this problem a lot. nowadays I just remote my Emacs
entirely, with x2go, so I dont have the problem anymore.

When I did have it I noticed some things:
- Running a local caching proxy dns helped a bit, but not entirely
- I could trick emacs into understanding the connection was gone, by:
  - killing connection
  - noticing which sockets were hanging with netstat
  - making new trick interfaces with the same IP:s as the hanging remote

This used to work, but was tedious.

I notice the same sort of issues with NFS mounts also. There are various
flags for NFS mounts that supposedly help, but not for me. The Linux
kernel network stack doesnt seem designed with mobile networking in
mind.

So, to summarize, x2go was the best thing I did in a long time :) (not
that x2go is flawless, either)

-- 
Joakim Verona

Re: Determining whether a TCP connection is up

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Weirdly enough, Gnus is the only one to show such issues, whereas I
> can't recall ever having this problem in mutt nor seeing it in Evolution
> or Thunderbird. And I don't believe in magic.

My guess would be that they implement timeouts.  Which is the tedious
work I had hoped not to have to do.

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php
  and http://lars.ingebrigtsen.no/2013/08/twenty-years-of-september.html

Re: Determining whether a TCP connection is up

[Stefan Monnier]

>> Weirdly enough, Gnus is the only one to show such issues, whereas I
>> can't recall ever having this problem in mutt nor seeing it in Evolution
>> or Thunderbird. And I don't believe in magic.
> My guess would be that they implement timeouts.  Which is the tedious
> work I had hoped not to have to do.

Another option is to kill connections after some time of inactivity,
rather than leave them open and suffer a timeout next time we need to
use it.


        Stefan

Re: Determining whether a TCP connection is up

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Another option is to kill connections after some time of inactivity,
> rather than leave them open and suffer a timeout next time we need to
> use it.

Yes.  But it's not very nice for the servers.  NNTP and IMAP servers do
the access control stuff upon connection, which can be
resource-intensive.

The earliest version of the ... Mozilla? or whatever they were called at
the time? (I think) ... news reader famously was written in a
connection-less method which made a news connection for every action it
made.  It DDoS-ed several NNTP servers.

Not that Gnus would have that effect.  >"?

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php
  and http://lars.ingebrigtsen.no/2013/08/twenty-years-of-september.html

Re: Determining whether a TCP connection is up

[Stefan Monnier]

>> Another option is to kill connections after some time of inactivity,
>> rather than leave them open and suffer a timeout next time we need to
>> use it.
> Yes.  But it's not very nice for the servers.  NNTP and IMAP servers do
> the access control stuff upon connection, which can be
> resource-intensive.

Those servers don't like to have thousands of idle connections tying up
resources either.
If you kill your connections after a reasonable amount of inactivity,
the cost of connection is not that bad.

> The earliest version of the ... Mozilla? or whatever they were called at
> the time? (I think) ... news reader famously was written in a
> connection-less method which made a news connection for every action it
> made.  It DDoS-ed several NNTP servers.

I think that "after some time of inactivity" is not quite the same as "a
news connection for every action".
E.g.: Use (process-put <proc> 'last-time (float-time)) in the process
filter, then use an idle timer to kill connections older than some
limit.  And when sending a command, check that last-time stamp, and
consider the process dead if it's older than the limit (in case the
idle-timer didn't do its job (yet) for some reason).


        Stefan