Re: Reviving Gnus after suspend/hibernation

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

Asking emacs-devel since the Gnus list didn't have any answers:

On Mon, 24 Oct 2011 09:23:09 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Tue, 18 Oct 2011 18:39:11 +0200 ludo@gnu.org (Ludovic Courtès) wrote: 
LC> When Gnus is left plugged or has non-agentized nnimap groups, recovering
LC> from suspend-to-RAM or hibernation has always been a problem for me:
LC> sometimes Emacs is frozen upon resume, trying to get data from some IMAP
LC> stream.
...
TZ> (Assuming modern GNU/Linux system is the main focus based on your
TZ> commands)

TZ> Is there a D-BUS signal for this, and can Emacs catch it?  If so I could
TZ> try to close the open connections in that handler.

TZ> What happens on a W32 system?

Any help is appreciated.  I don't know anything about these system events.

Thanks
Ted

Re: Reviving Gnus after suspend/hibernation

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Oct 2011 13:41:43 -0400
> Cc: ding@gnus.org
> 
> Asking emacs-devel since the Gnus list didn't have any answers:
> 
> On Mon, 24 Oct 2011 09:23:09 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
> 
> TZ> On Tue, 18 Oct 2011 18:39:11 +0200 ludo@gnu.org (Ludovic Courtès) wrote: 
> LC> When Gnus is left plugged or has non-agentized nnimap groups, recovering
> LC> from suspend-to-RAM or hibernation has always been a problem for me:
> LC> sometimes Emacs is frozen upon resume, trying to get data from some IMAP
> LC> stream.
> ...
> TZ> (Assuming modern GNU/Linux system is the main focus based on your
> TZ> commands)
> 
> TZ> Is there a D-BUS signal for this, and can Emacs catch it?  If so I could
> TZ> try to close the open connections in that handler.
> 
> TZ> What happens on a W32 system?
> 
> Any help is appreciated.  I don't know anything about these system events.

What exactly is the question?

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Thu, 27 Oct 2011 20:14:58 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 27 Oct 2011 13:41:43 -0400
>> Cc: ding@gnus.org
>> 
>> Asking emacs-devel since the Gnus list didn't have any answers:
>> 
>> On Mon, 24 Oct 2011 09:23:09 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>> 
TZ> On Tue, 18 Oct 2011 18:39:11 +0200 ludo@gnu.org (Ludovic Courtès) wrote: 
LC> When Gnus is left plugged or has non-agentized nnimap groups, recovering
LC> from suspend-to-RAM or hibernation has always been a problem for me:
LC> sometimes Emacs is frozen upon resume, trying to get data from some IMAP
LC> stream.
>> ...
TZ> (Assuming modern GNU/Linux system is the main focus based on your
TZ> commands)
>> 
TZ> Is there a D-BUS signal for this, and can Emacs catch it?  If so I could
TZ> try to close the open connections in that handler.
>> 
TZ> What happens on a W32 system?
>> 
>> Any help is appreciated.  I don't know anything about these system events.

EZ> What exactly is the question?

Sorry, I cut too much context.

Active Emacs network connections are not always closed properly upon
resuming from suspend.  This is a problem with GnuTLS, which keeps
waiting for data and hangs Emacs.  So I want to write a resume handler
that will close all the GnuTLS connections.

On Linux I think this is done with D-BUS, but is there a way on W32 as
well?

Thanks
Ted

Re: Reviving Gnus after suspend/hibernation

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 27 Oct 2011 13:37:53 -0500
> 
> Active Emacs network connections are not always closed properly upon
> resuming from suspend.  This is a problem with GnuTLS, which keeps
> waiting for data and hangs Emacs.  So I want to write a resume handler
> that will close all the GnuTLS connections.
> 
> On Linux I think this is done with D-BUS, but is there a way on W32 as
> well?

When the system resumes from suspend, programs are sent a special
message.  Emacs currently ignores this message, but we could add code
to listen to it and do whatever you want it to do.

Re: Reviving Gnus after suspend/hibernation

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 27 Oct 2011 20:14:58 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 
>
>>> From: Ted Zlatanov <tzz@lifelogs.com>
>>> Date: Thu, 27 Oct 2011 13:41:43 -0400
>>> Cc: ding@gnus.org
>>> 
>>> Asking emacs-devel since the Gnus list didn't have any answers:
>>> 
>>> On Mon, 24 Oct 2011 09:23:09 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>>> 
> TZ> On Tue, 18 Oct 2011 18:39:11 +0200 ludo@gnu.org (Ludovic Courtès) wrote: 
> LC> When Gnus is left plugged or has non-agentized nnimap groups, recovering
> LC> from suspend-to-RAM or hibernation has always been a problem for me:
> LC> sometimes Emacs is frozen upon resume, trying to get data from some IMAP
> LC> stream.
>>> ...
> TZ> (Assuming modern GNU/Linux system is the main focus based on your
> TZ> commands)
>>> 
> TZ> Is there a D-BUS signal for this, and can Emacs catch it?  If so I could
> TZ> try to close the open connections in that handler.
>>> 
> TZ> What happens on a W32 system?
>>> 
>>> Any help is appreciated.  I don't know anything about these system events.
>
> EZ> What exactly is the question?
>
> Sorry, I cut too much context.
>
> Active Emacs network connections are not always closed properly upon
> resuming from suspend.  This is a problem with GnuTLS, which keeps
> waiting for data and hangs Emacs.  So I want to write a resume handler
> that will close all the GnuTLS connections.
>
> On Linux I think this is done with D-BUS, but is there a way on W32 as
> well?

For the record I have had this or a similar problem since
forever. (there are bugreports etc.) It doesn't have to be a TLS
connection either.

normally I close the gnus connections manually after resume or network
change. When I forget and Emacs hangs, I use this little script:

,----
| #/bin/sh
| `lsof -n|grep emacs|grep nntp|sed "s/.*TCP\ \\([^:]*\\):.*->\\([^:].*\\):.*/  export a=\\1 export b=\\2/"`
| echo $a $b
| ifconfig lo:1 $a
| ifconfig lo:2 $b
| echo press enter when emacs is alive
| read
| ifconfig lo:1 down
| ifconfig lo:2 down
`----

but I don't know what the equivalent is on windoze.

It would be cool if this could be solved properly. (maybe you are seeing
another problem but it does sound similar)


>
> Thanks
> Ted
>

-- 
Joakim Verona

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Thu, 27 Oct 2011 20:48:26 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Thu, 27 Oct 2011 13:37:53 -0500
>> 
>> Active Emacs network connections are not always closed properly upon
>> resuming from suspend.  This is a problem with GnuTLS, which keeps
>> waiting for data and hangs Emacs.  So I want to write a resume handler
>> that will close all the GnuTLS connections.
>> 
>> On Linux I think this is done with D-BUS, but is there a way on W32 as
>> well?

EZ> When the system resumes from suspend, programs are sent a special
EZ> message.  Emacs currently ignores this message, but we could add code
EZ> to listen to it and do whatever you want it to do.

Exactly.  I want to do that on Linux and W32 systems (maybe NS/Mac OS X
as well).  Can you or anyone else help me get started?

Thanks
Ted

Re: Reviving Gnus after suspend/hibernation

[Stefan Monnier]

> Active Emacs network connections are not always closed properly upon
> resuming from suspend.  This is a problem with GnuTLS, which keeps
> waiting for data and hangs Emacs.  So I want to write a resume handler
> that will close all the GnuTLS connections.

Writing a resume handler would just patch over the problem, because this
problem can also show up without suspend&resume (typically, you're
behind a NAT router, and you leave the connection idle for a while,
after which the NAT router decides to forget about this connection).

Presumably setting the `keepalive' bit on the TCP connection should help, but:
- we should also make sure that C-g works.
- we should time out (hopefully keepalive does it for us) after a while.
- we should figure out what other applications (e.g. other MUAs) do.
- we should make it easy for the user to kill the connection if it is
  hung (e.g. make C-g kill the connection).


        Stefan

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Thu, 27 Oct 2011 17:12:28 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Active Emacs network connections are not always closed properly upon
>> resuming from suspend.  This is a problem with GnuTLS, which keeps
>> waiting for data and hangs Emacs.  So I want to write a resume handler
>> that will close all the GnuTLS connections.

SM> Writing a resume handler would just patch over the problem, because this
SM> problem can also show up without suspend&resume (typically, you're
SM> behind a NAT router, and you leave the connection idle for a while,
SM> after which the NAT router decides to forget about this connection).

SM> Presumably setting the `keepalive' bit on the TCP connection should help, but:
SM> - we should also make sure that C-g works.
SM> - we should time out (hopefully keepalive does it for us) after a while.
SM> - we should make it easy for the user to kill the connection if it is
SM>   hung (e.g. make C-g kill the connection).

You're right, but on suspend&resume we should not have to wait for the
TCP connection to time out.  Let's assume it's closed in that specific
case and set the keepalive for general use.  

Is this just a GnuTLS problem?  Does any of the rest of Emacs have
issues with hung connections?

SM> - we should figure out what other applications (e.g. other MUAs) do.

I think just reopening the connection is reasonable, no?  What else
needs to be figured out?

Ted

Re: Reviving Gnus after suspend/hibernation

[Richard Stallman]

    Exactly.  I want to do that on Linux and W32 systems (maybe NS/Mac OS X
    as well).

Would you please call the first system GNU/Linux?
It is more GNU than Linux.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Thu, 27 Oct 2011 19:48:27 -0400 Richard Stallman <rms@gnu.org> wrote: 

RS>     Exactly.  I want to do that on Linux and W32 systems (maybe NS/Mac OS X
RS>     as well).

RS> Would you please call the first system GNU/Linux?
RS> It is more GNU than Linux.

At the time I was specifically interested in the Linux kernel's resume
event, hence the slip of the tongue.

Ted

Re: Reviving Gnus after suspend/hibernation

[Stefan Monnier]

> You're right, but on suspend&resume we should not have to wait for the
> TCP connection to time out.  Let's assume it's closed in that specific
> case and set the keepalive for general use.

It strikes me as a non-Emacs-specific problem, so maybe the OS should
kill its TCP connection between suspend and resume.

> Is this just a GnuTLS problem?  Does any of the rest of Emacs have
> issues with hung connections?

I've been suffering from it for many years.  I'm not sure if it
affects NNTP connections (it probably does) but it for sure affects
nnimap with gnutls-cli.

SM> - we should figure out what other applications (e.g. other MUAs) do.
> I think just reopening the connection is reasonable, no?  What else
> needs to be figured out?

As I said, I can't think of any reason why we should be the only ones to
face this problem, so it'd be good to know how others dealt with it.


        Stefan

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Thu, 27 Oct 2011 20:33:23 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> You're right, but on suspend&resume we should not have to wait for the
>> TCP connection to time out.  Let's assume it's closed in that specific
>> case and set the keepalive for general use.

SM> It strikes me as a non-Emacs-specific problem, so maybe the OS should
SM> kill its TCP connection between suspend and resume.

It probably does.

>> Is this just a GnuTLS problem?  Does any of the rest of Emacs have
>> issues with hung connections?

SM> I've been suffering from it for many years.  I'm not sure if it
SM> affects NNTP connections (it probably does) but it for sure affects
SM> nnimap with gnutls-cli.

That's a completely different use case, you're wrapping a process with
arbitrary output when you use gnutls-cli.  I'd prefer to work just on
hung TCP connections.

Ted

Re: Reviving Gnus after suspend/hibernation

[Stefan Monnier]

>>> You're right, but on suspend&resume we should not have to wait for the
>>> TCP connection to time out.  Let's assume it's closed in that specific
>>> case and set the keepalive for general use.
SM> It strikes me as a non-Emacs-specific problem, so maybe the OS should
SM> kill its TCP connection between suspend and resume.
> It probably does.

The Linux kernel definitely does not (and barring a NAT router, or
a resume with a different IP, or attempted communication on the link
while you're sleeping, the TCP connection will be faithfully waiting
for us when we resume).

>>> Is this just a GnuTLS problem?  Does any of the rest of Emacs have
>>> issues with hung connections?
SM> I've been suffering from it for many years.  I'm not sure if it
SM> affects NNTP connections (it probably does) but it for sure affects
SM> nnimap with gnutls-cli.
> That's a completely different use case, you're wrapping a process with
> arbitrary output when you use gnutls-cli.  I'd prefer to work just on
> hung TCP connections.

There are different cases at play indeed: gnutls-cli is one, NNTP
(without TLS) is another, and nnimap with libgnutls is yet another.
I think I see similar problems in all three cases, caused by the same
underlying OS-level problem.  But since the C and Elisp code that wraps
it is different, the exact behavior will differ (e.g. typically w.r.t
reaction to C-g, possibility of a timeout, ...).


        Stefan

Re: Reviving Gnus after suspend/hibernation

[Jason Rumney]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The Linux kernel definitely does not (and barring a NAT router, or
> a resume with a different IP, or attempted communication on the link
> while you're sleeping, the TCP connection will be faithfully waiting
> for us when we resume).

The other end will close it after a timeout. Usually 5 minutes.  So you
end up with a socket to nowhere, and need to wait another 5 minutes on
the local end for the timeout to find out the socket is not
working. This isn't just a problem during suspend/resume, it also
happens if the network disappears suddenly.

But this isn't the problem I've experienced with Gnus after
suspend/resume. Gnus apparently knows the socket is closed, as it
very quickly reports the connection failure and starts working in
offline mode.  What is missing is the ability to automatically reconnect
and continue, without going back to the Groups buffer and pressing M-g
(which inhibits the offline mode, as opposed to g, which doesn't).

Re: Reviving Gnus after suspend/hibernation

[Stefan Monnier]

>> The Linux kernel definitely does not (and barring a NAT router, or
>> a resume with a different IP, or attempted communication on the link
>> while you're sleeping, the TCP connection will be faithfully waiting
>> for us when we resume).
> The other end will close it after a timeout.  Usually 5 minutes.

I can assure you that I've seen TCP connections without any packet
exchanged for a lot more than 5 minutes.  And indeed most NAT routers
wait longer than 5 minutes before considering a TCP connection as dead.

The timeout you're talking about only exists if the connection uses the
`tcp keepalive' feature (and yes, we want to use the tcp keepalive on
those connections, and I added code for that).

> But this isn't the problem I've experienced with Gnus after
> suspend/resume.  Gnus apparently knows the socket is closed, as it
> very quickly reports the connection failure and starts working in
> offline mode.  What is missing is the ability to automatically reconnect
> and continue, without going back to the Groups buffer and pressing M-g
> (which inhibits the offline mode, as opposed to g, which doesn't).

That's another issue ... ah yes: bug#9244


        Stefan

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Fri, 28 Oct 2011 20:50:24 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> You're right, but on suspend&resume we should not have to wait for the
>>>> TCP connection to time out.  Let's assume it's closed in that specific
>>>> case and set the keepalive for general use.
SM> It strikes me as a non-Emacs-specific problem, so maybe the OS should
SM> kill its TCP connection between suspend and resume.
>> It probably does.

SM> The Linux kernel definitely does not (and barring a NAT router, or
SM> a resume with a different IP, or attempted communication on the link
SM> while you're sleeping, the TCP connection will be faithfully waiting
SM> for us when we resume).

Please remember most of the Internet is behind a NAT router.  TCP
connections are very rarely around for long.

>>>> Is this just a GnuTLS problem?  Does any of the rest of Emacs have
>>>> issues with hung connections?
SM> I've been suffering from it for many years.  I'm not sure if it
SM> affects NNTP connections (it probably does) but it for sure affects
SM> nnimap with gnutls-cli.
>> That's a completely different use case, you're wrapping a process with
>> arbitrary output when you use gnutls-cli.  I'd prefer to work just on
>> hung TCP connections.

SM> There are different cases at play indeed: gnutls-cli is one, NNTP
SM> (without TLS) is another, and nnimap with libgnutls is yet another.
SM> I think I see similar problems in all three cases, caused by the same
SM> underlying OS-level problem.  But since the C and Elisp code that wraps
SM> it is different, the exact behavior will differ (e.g. typically w.r.t
SM> reaction to C-g, possibility of a timeout, ...).

Right.  I will just take a look at the GnuTLS code and maybe TCP in
general, and hope others find the hung processes interesting.

Ted

Re: Reviving Gnus after suspend/hibernation

[Ted Zlatanov]

On Sat, 29 Oct 2011 12:13:04 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> The timeout you're talking about only exists if the connection uses the
SM> `tcp keepalive' feature (and yes, we want to use the tcp keepalive on
SM> those connections, and I added code for that).

Can you give a rev number or something to help me find what you did?  I
don't see it in the recent Bazaar logs, unless you mean

revno: 104174
committer: Stefan Monnier <monnier@iro.umontreal.ca>
branch nick: trunk
timestamp: Mon 2011-05-09 16:41:14 -0300
message:
  * lisp/gnus/nntp.el (nntp-open-connection): Set TCP keepalive option.

which is just for NNTP connections.

Thanks
Ted

Re: Reviving Gnus after suspend/hibernation

[Antoine Levitt]

27/10/11 19:41, Ted Zlatanov
> Asking emacs-devel since the Gnus list didn't have any answers:
>
> On Mon, 24 Oct 2011 09:23:09 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 
>
> TZ> On Tue, 18 Oct 2011 18:39:11 +0200 ludo@gnu.org (Ludovic Courtès) wrote: 
> LC> When Gnus is left plugged or has non-agentized nnimap groups, recovering
> LC> from suspend-to-RAM or hibernation has always been a problem for me:
> LC> sometimes Emacs is frozen upon resume, trying to get data from some IMAP
> LC> stream.
> ...
> TZ> (Assuming modern GNU/Linux system is the main focus based on your
> TZ> commands)
>
> TZ> Is there a D-BUS signal for this, and can Emacs catch it?  If so I could
> TZ> try to close the open connections in that handler.
>
> TZ> What happens on a W32 system?
>
> Any help is appreciated.  I don't know anything about these system events.
>
> Thanks
> Ted

Just a data point: I used to have tons of issues like this one (and
basically used to quit and run gnus again every time I resumed from
suspend). I no longer have any. I _think_ it's because I switched from
using a shell connection to a local TCP connection. Also, I used to get
random hangs, even with a TCP connection, and they disappeared magically
a couple of months ago. I don't know who did that, but good job :-)

Re: Reviving Gnus after suspend/hibernation

[Stefan Monnier]

> Can you give a rev number or something to help me find what you did?  I
> don't see it in the recent Bazaar logs, unless you mean

> revno: 104174
> committer: Stefan Monnier <monnier@iro.umontreal.ca>
> branch nick: trunk
> timestamp: Mon 2011-05-09 16:41:14 -0300
> message:
>   * lisp/gnus/nntp.el (nntp-open-connection): Set TCP keepalive option.

> which is just for NNTP connections.

Ah, indeed, that's the one.  So it should maybe be moved elsewhere (or
copied) so it applies in other cases as well.


        Stefan

Re: Reviving Gnus after suspend/hibernation

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ah, indeed, that's the one.  So it should maybe be moved elsewhere (or
> copied) so it applies in other cases as well.

I think it would make sense to have all networks connections switch on
TCP keepalive by default.  (And we might provide a user customisation to
switch it off, if needed, but I doubt it's needed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Reviving Gnus after suspend/hibernation

[Stefan Monnier]

>> Ah, indeed, that's the one.  So it should maybe be moved elsewhere (or
>> copied) so it applies in other cases as well.

> I think it would make sense to have all networks connections switch on
> TCP keepalive by default.  (And we might provide a user customisation to
> switch it off, if needed, but I doubt it's needed.)

I've just added it to nnimap.el.  If you're suggesting we do it for all
connections in Emacs (rather than only in Gnus), then I'd rather wait
for 24.2 before doing such a thing (I'm not even sure if it's a good
idea).


        Stefan

Re: Reviving Gnus after suspend/hibernation

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I've just added it to nnimap.el.  If you're suggesting we do it for all
> connections in Emacs (rather than only in Gnus), then I'd rather wait
> for 24.2 before doing such a thing

Sure.

> (I'm not even sure if it's a good idea).

Me neither.  But at the moment I can't really think if any major
drawbacks.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/