bug#16026: Gnus shouldn't use old connections

bug#16026: Gnus shouldn't use old connections

[Sebastien Vauban]

Stefan Monnier wrote:
> Package: Emacs
> Version: 24.3.50
>
> Gnus should consider a connection as dead if it has been idle for more
> than some timeout value (e.g. 5 minutes).  This bites me every day when
> I wake up my computer: if I just hit `g' to refresh the *Group* buffer,
> it *always* hangs (I guess it would timeout but only after a long wait,
> like 2h).  Instead, I have to go to *Server*, close each server, go back
> to *Group* and then hit `g'.

Weird. I do the same as you, or almost [1], and don't have that problem.

Best regards,
  Seb

[1] When I'm closing the lid, the computer is put to "sleep" (not "hibernate").
And, every night, I switch the mains off for the computer room.

-- 
Sebastien Vauban

bug#16026: Gnus shouldn't use old connections

[Stefan Monnier]

>> Gnus should consider a connection as dead if it has been idle for more
>> than some timeout value (e.g. 5 minutes).  This bites me every day when
>> I wake up my computer: if I just hit `g' to refresh the *Group* buffer,
>> it *always* hangs (I guess it would timeout but only after a long wait,
>> like 2h).  Instead, I have to go to *Server*, close each server, go back
>> to *Group* and then hit `g'.
> Weird. I do the same as you, or almost [1], and don't have that problem.

Of course, the occurrence of my particular problem depends on various
factors, such as whether you get the same IP address when you wake up,
whether you're in the same network when you wake up, whether you're
behind a NAT, how long the NAT router remembers connections, whether the
NAT router's own IP has changed, ...


        Stefan

bug#16026: Gnus shouldn't use old connections

[Ted Zlatanov]

On Mon, 02 Dec 2013 11:12:20 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>> Gnus should consider a connection as dead if it has been idle for more
>>> than some timeout value (e.g. 5 minutes).  This bites me every day when
>>> I wake up my computer: if I just hit `g' to refresh the *Group* buffer,
>>> it *always* hangs (I guess it would timeout but only after a long wait,
>>> like 2h).  Instead, I have to go to *Server*, close each server, go back
>>> to *Group* and then hit `g'.
>> Weird. I do the same as you, or almost [1], and don't have that problem.

SM> Of course, the occurrence of my particular problem depends on various
SM> factors, such as whether you get the same IP address when you wake up,
SM> whether you're in the same network when you wake up, whether you're
SM> behind a NAT, how long the NAT router remembers connections, whether the
SM> NAT router's own IP has changed, ...

In the Gnus mailing list we discussed an easy solution (just close all
server connections before `g') but providing that as a user-level
defcustom feels like giving up.  See http://thread.gmane.org/gmane.emacs.gnus.general/80262/focus=83843

I suggested waiting for Lars' opinion.

Ted

bug#16026: Gnus shouldn't use old connections

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 02 Dec 2013 11:12:20 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
>>>> Gnus should consider a connection as dead if it has been idle for more
>>>> than some timeout value (e.g. 5 minutes).  This bites me every day when
>>>> I wake up my computer: if I just hit `g' to refresh the *Group* buffer,
>>>> it *always* hangs (I guess it would timeout but only after a long wait,
>>>> like 2h).  Instead, I have to go to *Server*, close each server, go back
>>>> to *Group* and then hit `g'.

[...]

> In the Gnus mailing list we discussed an easy solution (just close all
> server connections before `g') but providing that as a user-level
> defcustom feels like giving up.  See http://thread.gmane.org/gmane.emacs.gnus.general/80262/focus=83843
>
> I suggested waiting for Lars' opinion.

There is `nntp-connection-timeout', and nnimap could have something
similar, I guess.

The difficulty is in determining what a good timeout should be.  If you
send an EXPUNGE to an IMAP server, it may legitimately take many
seconds, and you get no output from the server while it's doing this.
If you kill the connection and reissue the commands, you then get error
messages, I think?

But issuing EXPUNGE as the first thing after you haven't used a
connection in a while is unusual.  What usually happens is that you
enter an IMAP group, read a few messages, then your ISP decides that you
should change your IP number, and then you hit `q' which will normally
issue a marks updating command.  That should normally give a response
within a reasonable time.

If a command like that takes more than a few seconds, then we should
probably reconnect...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#16026: Gnus shouldn't use old connections

[Stefan Monnier]

> There is `nntp-connection-timeout', and nnimap could have something
> similar, I guess.

> The difficulty is in determining what a good timeout should be.  If you
> send an EXPUNGE to an IMAP server, it may legitimately take many
> seconds, and you get no output from the server while it's doing this.
> If you kill the connection and reissue the commands, you then get error
> messages, I think?

I think in my case a timeout of 30min would cover the main cases already
(i.e. the machine was suspended for more than 30min).


        Stefan

bug#16026: Gnus shouldn't use old connections

[Ted Zlatanov]

On Wed, 25 Jan 2017 13:26:04 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> There is `nntp-connection-timeout', and nnimap could have something
>> similar, I guess.

>> The difficulty is in determining what a good timeout should be.  If you
>> send an EXPUNGE to an IMAP server, it may legitimately take many
>> seconds, and you get no output from the server while it's doing this.
>> If you kill the connection and reissue the commands, you then get error
>> messages, I think?

SM> I think in my case a timeout of 30min would cover the main cases already
SM> (i.e. the machine was suspended for more than 30min).

I would make it a per-connection setting using the scheme that Michael
Albinus has implemented. Then it can be set per host etc.

Ted

bug#16026: Gnus shouldn't use old connections

[Stefan Monnier]

> I would make it a per-connection setting using the scheme that Michael
> Albinus has implemented. Then it can be set per host etc.

Not sure if it's worth the trouble, but as long as I can easily set it
globally, that's fine by me.


        Stefan


PS: For me, the need is global because it is mostly caused by NAT
translation, which hence affects the connections to basically all hosts,
except for the very few which are on the same side of the firewall as
I am.

bug#16026: Gnus shouldn't use old connections

[Ted Zlatanov]

On Mon, 30 Jan 2017 17:38:22 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> I would make it a per-connection setting using the scheme that Michael
>> Albinus has implemented. Then it can be set per host etc.

SM> Not sure if it's worth the trouble, but as long as I can easily set it
SM> globally, that's fine by me.

SM> PS: For me, the need is global because it is mostly caused by NAT
SM> translation, which hence affects the connections to basically all hosts,
SM> except for the very few which are on the same side of the firewall as
SM> I am.

Yes, that's a common situation. Another is home vs. work connections with
different profiles.

It's easy to create an ad-hoc system just for network timeout
parameters, like I did with many GnuTLS parameters early on, but the
systematic approach Michael gave us is much better. The only reason I
haven't implemented it is lack of time.

Ted

bug#16026: Gnus shouldn't use old connections

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> It's easy to create an ad-hoc system just for network timeout
> parameters, like I did with many GnuTLS parameters early on, but the
> systematic approach Michael gave us is much better. The only reason I
> haven't implemented it is lack of time.

If I could help, just ping me. Not that I have too much free time, but
I'm interested in this.

> Ted

Best regards, Michael.

bug#16026: Connection specific settings and proxies

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

>> It's easy to create an ad-hoc system just for network timeout
>> parameters, like I did with many GnuTLS parameters early on, but the
>> systematic approach Michael gave us is much better. The only reason I
>> haven't implemented it is lack of time.
>
> If I could help, just ping me. Not that I have too much free time, but
> I'm interested in this.

I was just pondering something that's perhaps related: HTTP proxying in
the with-url branch.

url.el uses an alist on the form ((("http" . "foo.bar:port")
"proxy:port")) or something to determine what connections should be
proxied.  It doesn't support HTTPS proxies (i.e., "CONNECT"), so it's
rather underspecified, if I read the code correctly.

I'm also not sure what level to put the proxies on: Should they be
passed in from the application (i.e., eww, Gnus, etc) or should it be a
global Emacs setting?

If the latter, you could see having a call like

(set-proxy :match-domain ".*\\.foo\\.bar\\'"
           :target-port 443
           :proxy-server "localhost"
           :proxy-port 80
           :method 'connect)

to set up the proxy.  (Well, really, you'd have a mode that would allow
you to tweak the global proxy setup.)

Now, this all kinda ties in with what Ted was talking about: A
per-connection setting thing.  So you could envision doing just the same
with, say, IMAP, if you need to have that be proxied or something, and
then it turns out that the right layer for this stuff is probably in
`open-network-stream': Almost all networking already goes through that
function, so by implementing it there, all clients get automatic proxy
support.

The same mechanism for determining things like lifecycle settings (i.e.,
network timeout parameters) could somehow be squeezed in here.

But I've read the doc for connection-local-set-class-variables, and I
don't quite see how to match up these needs with that mechanism...

And, like I said, I don't know whether it's the right design choice to
have these settings be global, or whether they should be passed in
explicitly from each application.  Would users want to use one set of
proxies while reading HTML news from Gnus and another when reading from
eww?  Perhaps?  Perhaps not?

So many options...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#16026: Connection specific settings and proxies

[Ted Zlatanov]

On Thu, 02 Feb 2017 04:43:10 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> I'm also not sure what level to put the proxies on: Should they be
LI> passed in from the application (i.e., eww, Gnus, etc) or should it be a
LI> global Emacs setting?

LI> If the latter, you could see having a call like

LI> (set-proxy :match-domain ".*\\.foo\\.bar\\'"
LI>            :target-port 443
LI>            :proxy-server "localhost"
LI>            :proxy-port 80
LI>            :method 'connect)

LI> to set up the proxy.  (Well, really, you'd have a mode that would allow
LI> you to tweak the global proxy setup.)

The way Michael set it up, you have connection profiles ("classes") that
can be associated with any variables. (For passers-by, see
`connection-local-set-class-variables', `connection-local-set-classes',
and `with-connection-local-classes'). Profiles can be overlaid to
augment each other.

(Michael: maybe it's not too late to change "class" to "profile"
because the former is so overloaded in our field?)

LI> And, like I said, I don't know whether it's the right design choice to
LI> have these settings be global, or whether they should be passed in
LI> explicitly from each application.  Would users want to use one set of
LI> proxies while reading HTML news from Gnus and another when reading from
LI> eww?  Perhaps?  Perhaps not?

I see. I think the classes should be associated with applications and
protocols and login names, not just connections. Michael, what do you
think? That would require changing the identification parameter to
`connection-local-get-classes' to be an alist or a plist like
:user U :application X :protocol Y :machine Z

I can imagine more criteria in the future, so the identification should
be flexible. This has a pretty big semantic overlap with how auth-source
selects credentials and NSM applies security polity, so I think it makes
sense to absorb those things into the same hierarchy. So IMO this is a
chance to consolidate a lot of disparate code and improve the user
experience.

Then you could have:

"class C1(contains the proxy P1) for IMAP to machines A B C"
"class C2(contains the proxy P2) for everything"
"class C3(contains the proxy P3) for machines B C D"

So IMAP to machine B would get C1/P1, but HTTP to machine B would get
C3/P3 (all the matching classes should be applied in specificity order
so the most specific one wins).

Not all parameters apply everywhere, so we're looking to extract the
truly global ones first: network timeout, proxies, NSM policy, GnuTLS
parameters, auth-source entries.

Ted

bug#16026: Connection specific settings and proxies

[Daniel McClanahan]

On Wed, Feb 1, 2017 at 9:43 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> And, like I said, I don't know whether it's the right design choice to have
> these settings be global, or whether they should be passed in explicitly from
> each application.  Would users want to use one set of proxies while reading
> HTML news from Gnus and another when reading from eww?  Perhaps?  Perhaps not?

I'm no expert, but I've often been frustrated by configuration options that
aren't exposed as defcustoms. I think a defcustom of an alist mapping the match
domain to a list/plist of proxy settings would work. If `open-network-stream'
only accepts proxy settings as an argument and Gnus/eww/etc wants to allow
user-specified proxies, it seems each would have to implement its own defcustom
(or some other method) to handle that anyway; configuring these for each package
seems annoying. Callers can use a let binding if they want to override it, which
is about as easy as an argument.

A single global value would make it difficult to have different settings for
different callers. One potential solution is to have an optional "tag" symbol as
part of the proxy settings, which would only use the proxy if the caller
specified the same tag, maybe as a buffer-local variable. So Gnus could set the
tag to 'gnus or something in its mode, and a user could add a proxy entry with
the 'gnus tag. This might be confusing, though.

I don't use proxies too extensively, but something like that would have removed
the need for some hacky defadvice I used to configure my ssh connections on a
corporate network last summer. Just a thought.

bug#16026: Connection specific settings and proxies

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> The way Michael set it up, you have connection profiles ("classes") that
> can be associated with any variables. (For passers-by, see
> `connection-local-set-class-variables', `connection-local-set-classes',
> and `with-connection-local-classes'). Profiles can be overlaid to
> augment each other.
>
> (Michael: maybe it's not too late to change "class" to "profile"
> because the former is so overloaded in our field?)

Well, I've stolen the "class" name from directory-local variables. The
concept of connection-local variables is closed to that of directory-local.

No problem to rename it, if it helps. And sometimes I'm even not sure
whether we need the class/profile concept; it might be even sufficient
to bind variables to connections directly, like variables can be bound
directly to buffers.

> LI> And, like I said, I don't know whether it's the right design choice to
> LI> have these settings be global, or whether they should be passed in
> LI> explicitly from each application.  Would users want to use one set of
> LI> proxies while reading HTML news from Gnus and another when reading from
> LI> eww?  Perhaps?  Perhaps not?
>
> I see. I think the classes should be associated with applications and
> protocols and login names, not just connections. Michael, what do you
> think? That would require changing the identification parameter to
> `connection-local-get-classes' to be an alist or a plist like
> :user U :application X :protocol Y :machine Z

Well, currently a class is bound to a connection via a regexp, which is
used for checking the result of `file-remote-p'. This is good for Tramp,
but it is less good for url-* functionality unless `url-handler-mode' is
enabled, and it does not work for other applications like imap-based
connections.

Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
(or also a regexp?). All of them shall be optional.

> Ted

Best regards, Michael.

bug#16026: Connection specific settings and proxies

[Ted Zlatanov]

On Fri, 03 Feb 2017 16:37:52 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> Hi Ted,

>> The way Michael set it up, you have connection profiles ("classes") that
>> can be associated with any variables. (For passers-by, see
>> `connection-local-set-class-variables', `connection-local-set-classes',
>> and `with-connection-local-classes'). Profiles can be overlaid to
>> augment each other.
>> 
>> (Michael: maybe it's not too late to change "class" to "profile"
>> because the former is so overloaded in our field?)

MA> Well, I've stolen the "class" name from directory-local variables. The
MA> concept of connection-local variables is closed to that of directory-local.

MA> No problem to rename it, if it helps. And sometimes I'm even not sure
MA> whether we need the class/profile concept; it might be even sufficient
MA> to bind variables to connections directly, like variables can be bound
MA> directly to buffers.

I think from the user's side, it's necessary. We have to associate the
data with a profile, which is then exposed as a configurable entity.

Let's call them "connection profiles" to be very clear, since just
"profile" can be ambiguous as well.

>> I see. I think the classes should be associated with applications and
>> protocols and login names, not just connections. Michael, what do you
>> think? That would require changing the identification parameter to
>> `connection-local-get-classes' to be an alist or a plist like
>> :user U :application X :protocol Y :machine Z

MA> Well, currently a class is bound to a connection via a regexp, which is
MA> used for checking the result of `file-remote-p'. This is good for Tramp,
MA> but it is less good for url-* functionality unless `url-handler-mode' is
MA> enabled, and it does not work for other applications like imap-based
MA> connections.

MA> Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
MA> (or also a regexp?). All of them shall be optional.

Wonderful. I propose that the application should be a defgroup symbol,
and every connection profile should be required to have a defgroup.

That would give us an implicit connection profile hierarchy and a place
to hook connection profile customizations at the UI level.

So defining connection profiles would be something like this:

(add-connection-profiles :user U :defgroup X :protocol Y :machine Z
                         :profile 'UXYZ :vars '(...))

(add-connection-profiles :profile 'general :vars '(...))

And then the application Gnus would do (note it's required that the
defgroup 'gnus exists):

(connection-profile-get-local-variables :defgroup 'gnus ...other criteria...)

From the user's side, they would say

M-x customize-connection-profile UXYZ ; change criteria or settings
M-x customize-connection-profile general ; change criteria or settings
M-x create-connection-profile atwork ; specify desired settings
M-x connection-profiles ; some kind of table/list UI

Let me know what you think.

Thanks
Ted

bug#16026: Connection specific settings and proxies

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> Well, I've stolen the "class" name from directory-local variables. The
> MA> concept of connection-local variables is closed to that of directory-local.
>
> MA> No problem to rename it, if it helps. And sometimes I'm even not sure
> MA> whether we need the class/profile concept; it might be even sufficient
> MA> to bind variables to connections directly, like variables can be bound
> MA> directly to buffers.
>
> I think from the user's side, it's necessary. We have to associate the
> data with a profile, which is then exposed as a configurable entity.
>
> Let's call them "connection profiles" to be very clear, since just
> "profile" can be ambiguous as well.

OK, I'll change next time I'll touch the code.

> MA> Well, currently a class is bound to a connection via a regexp, which is
> MA> used for checking the result of `file-remote-p'. This is good for Tramp,
> MA> but it is less good for url-* functionality unless `url-handler-mode' is
> MA> enabled, and it does not work for other applications like imap-based
> MA> connections.
>
> MA> Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
> MA> (or also a regexp?). All of them shall be optional.
>
> Wonderful. I propose that the application should be a defgroup symbol,
> and every connection profile should be required to have a defgroup.
>
> That would give us an implicit connection profile hierarchy and a place
> to hook connection profile customizations at the UI level.
>
> So defining connection profiles would be something like this:
>
> (add-connection-profiles :user U :defgroup X :protocol Y :machine Z
>                          :profile 'UXYZ :vars '(...))
>
> (add-connection-profiles :profile 'general :vars '(...))

Well, I'm a little bit lost. What does the defgroup here? I understand
that replaces what you have called initially an application, but I don't
see what defcustoms shall be placed now under the defgroup
hierarchy. Could you pls give an example?

I also know that there are users reluctant to the customization
machinery. Wouldn't we loose them?

> Thanks
> Ted

Best regards, Michael.

bug#16026: Connection specific settings and proxies

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

>> So defining connection profiles would be something like this:
>>
>> (add-connection-profiles :user U :defgroup X :protocol Y :machine Z
>>                          :profile 'UXYZ :vars '(...))
>>
>> (add-connection-profiles :profile 'general :vars '(...))
>
> Well, I'm a little bit lost. What does the defgroup here? I understand
> that replaces what you have called initially an application, but I don't
> see what defcustoms shall be placed now under the defgroup
> hierarchy. Could you pls give an example?
>
> I also know that there are users reluctant to the customization
> machinery. Wouldn't we loose them?

Yeah, I think a special mode to tweak connection specific variables
would be better, perhaps...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no