bug#16026: Connection specific settings and proxies

bug#16026: Connection specific settings and proxies

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

>> It's easy to create an ad-hoc system just for network timeout
>> parameters, like I did with many GnuTLS parameters early on, but the
>> systematic approach Michael gave us is much better. The only reason I
>> haven't implemented it is lack of time.
>
> If I could help, just ping me. Not that I have too much free time, but
> I'm interested in this.

I was just pondering something that's perhaps related: HTTP proxying in
the with-url branch.

url.el uses an alist on the form ((("http" . "foo.bar:port")
"proxy:port")) or something to determine what connections should be
proxied.  It doesn't support HTTPS proxies (i.e., "CONNECT"), so it's
rather underspecified, if I read the code correctly.

I'm also not sure what level to put the proxies on: Should they be
passed in from the application (i.e., eww, Gnus, etc) or should it be a
global Emacs setting?

If the latter, you could see having a call like

(set-proxy :match-domain ".*\\.foo\\.bar\\'"
           :target-port 443
           :proxy-server "localhost"
           :proxy-port 80
           :method 'connect)

to set up the proxy.  (Well, really, you'd have a mode that would allow
you to tweak the global proxy setup.)

Now, this all kinda ties in with what Ted was talking about: A
per-connection setting thing.  So you could envision doing just the same
with, say, IMAP, if you need to have that be proxied or something, and
then it turns out that the right layer for this stuff is probably in
`open-network-stream': Almost all networking already goes through that
function, so by implementing it there, all clients get automatic proxy
support.

The same mechanism for determining things like lifecycle settings (i.e.,
network timeout parameters) could somehow be squeezed in here.

But I've read the doc for connection-local-set-class-variables, and I
don't quite see how to match up these needs with that mechanism...

And, like I said, I don't know whether it's the right design choice to
have these settings be global, or whether they should be passed in
explicitly from each application.  Would users want to use one set of
proxies while reading HTML news from Gnus and another when reading from
eww?  Perhaps?  Perhaps not?

So many options...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: bug#16026: Connection specific settings and proxies

[Ted Zlatanov]

On Thu, 02 Feb 2017 04:43:10 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> I'm also not sure what level to put the proxies on: Should they be
LI> passed in from the application (i.e., eww, Gnus, etc) or should it be a
LI> global Emacs setting?

LI> If the latter, you could see having a call like

LI> (set-proxy :match-domain ".*\\.foo\\.bar\\'"
LI>            :target-port 443
LI>            :proxy-server "localhost"
LI>            :proxy-port 80
LI>            :method 'connect)

LI> to set up the proxy.  (Well, really, you'd have a mode that would allow
LI> you to tweak the global proxy setup.)

The way Michael set it up, you have connection profiles ("classes") that
can be associated with any variables. (For passers-by, see
`connection-local-set-class-variables', `connection-local-set-classes',
and `with-connection-local-classes'). Profiles can be overlaid to
augment each other.

(Michael: maybe it's not too late to change "class" to "profile"
because the former is so overloaded in our field?)

LI> And, like I said, I don't know whether it's the right design choice to
LI> have these settings be global, or whether they should be passed in
LI> explicitly from each application.  Would users want to use one set of
LI> proxies while reading HTML news from Gnus and another when reading from
LI> eww?  Perhaps?  Perhaps not?

I see. I think the classes should be associated with applications and
protocols and login names, not just connections. Michael, what do you
think? That would require changing the identification parameter to
`connection-local-get-classes' to be an alist or a plist like
:user U :application X :protocol Y :machine Z

I can imagine more criteria in the future, so the identification should
be flexible. This has a pretty big semantic overlap with how auth-source
selects credentials and NSM applies security polity, so I think it makes
sense to absorb those things into the same hierarchy. So IMO this is a
chance to consolidate a lot of disparate code and improve the user
experience.

Then you could have:

"class C1(contains the proxy P1) for IMAP to machines A B C"
"class C2(contains the proxy P2) for everything"
"class C3(contains the proxy P3) for machines B C D"

So IMAP to machine B would get C1/P1, but HTTP to machine B would get
C3/P3 (all the matching classes should be applied in specificity order
so the most specific one wins).

Not all parameters apply everywhere, so we're looking to extract the
truly global ones first: network timeout, proxies, NSM policy, GnuTLS
parameters, auth-source entries.

Ted

Re: bug#16026: Connection specific settings and proxies

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> The way Michael set it up, you have connection profiles ("classes") that
> can be associated with any variables. (For passers-by, see
> `connection-local-set-class-variables', `connection-local-set-classes',
> and `with-connection-local-classes'). Profiles can be overlaid to
> augment each other.
>
> (Michael: maybe it's not too late to change "class" to "profile"
> because the former is so overloaded in our field?)

Well, I've stolen the "class" name from directory-local variables. The
concept of connection-local variables is closed to that of directory-local.

No problem to rename it, if it helps. And sometimes I'm even not sure
whether we need the class/profile concept; it might be even sufficient
to bind variables to connections directly, like variables can be bound
directly to buffers.

> LI> And, like I said, I don't know whether it's the right design choice to
> LI> have these settings be global, or whether they should be passed in
> LI> explicitly from each application.  Would users want to use one set of
> LI> proxies while reading HTML news from Gnus and another when reading from
> LI> eww?  Perhaps?  Perhaps not?
>
> I see. I think the classes should be associated with applications and
> protocols and login names, not just connections. Michael, what do you
> think? That would require changing the identification parameter to
> `connection-local-get-classes' to be an alist or a plist like
> :user U :application X :protocol Y :machine Z

Well, currently a class is bound to a connection via a regexp, which is
used for checking the result of `file-remote-p'. This is good for Tramp,
but it is less good for url-* functionality unless `url-handler-mode' is
enabled, and it does not work for other applications like imap-based
connections.

Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
(or also a regexp?). All of them shall be optional.

> Ted

Best regards, Michael.

Re: bug#16026: Connection specific settings and proxies

[Ted Zlatanov]

On Fri, 03 Feb 2017 16:37:52 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> Hi Ted,

>> The way Michael set it up, you have connection profiles ("classes") that
>> can be associated with any variables. (For passers-by, see
>> `connection-local-set-class-variables', `connection-local-set-classes',
>> and `with-connection-local-classes'). Profiles can be overlaid to
>> augment each other.
>> 
>> (Michael: maybe it's not too late to change "class" to "profile"
>> because the former is so overloaded in our field?)

MA> Well, I've stolen the "class" name from directory-local variables. The
MA> concept of connection-local variables is closed to that of directory-local.

MA> No problem to rename it, if it helps. And sometimes I'm even not sure
MA> whether we need the class/profile concept; it might be even sufficient
MA> to bind variables to connections directly, like variables can be bound
MA> directly to buffers.

I think from the user's side, it's necessary. We have to associate the
data with a profile, which is then exposed as a configurable entity.

Let's call them "connection profiles" to be very clear, since just
"profile" can be ambiguous as well.

>> I see. I think the classes should be associated with applications and
>> protocols and login names, not just connections. Michael, what do you
>> think? That would require changing the identification parameter to
>> `connection-local-get-classes' to be an alist or a plist like
>> :user U :application X :protocol Y :machine Z

MA> Well, currently a class is bound to a connection via a regexp, which is
MA> used for checking the result of `file-remote-p'. This is good for Tramp,
MA> but it is less good for url-* functionality unless `url-handler-mode' is
MA> enabled, and it does not work for other applications like imap-based
MA> connections.

MA> Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
MA> (or also a regexp?). All of them shall be optional.

Wonderful. I propose that the application should be a defgroup symbol,
and every connection profile should be required to have a defgroup.

That would give us an implicit connection profile hierarchy and a place
to hook connection profile customizations at the UI level.

So defining connection profiles would be something like this:

(add-connection-profiles :user U :defgroup X :protocol Y :machine Z
                         :profile 'UXYZ :vars '(...))

(add-connection-profiles :profile 'general :vars '(...))

And then the application Gnus would do (note it's required that the
defgroup 'gnus exists):

(connection-profile-get-local-variables :defgroup 'gnus ...other criteria...)

From the user's side, they would say

M-x customize-connection-profile UXYZ ; change criteria or settings
M-x customize-connection-profile general ; change criteria or settings
M-x create-connection-profile atwork ; specify desired settings
M-x connection-profiles ; some kind of table/list UI

Let me know what you think.

Thanks
Ted

Re: bug#16026: Connection specific settings and proxies

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> Well, I've stolen the "class" name from directory-local variables. The
> MA> concept of connection-local variables is closed to that of directory-local.
>
> MA> No problem to rename it, if it helps. And sometimes I'm even not sure
> MA> whether we need the class/profile concept; it might be even sufficient
> MA> to bind variables to connections directly, like variables can be bound
> MA> directly to buffers.
>
> I think from the user's side, it's necessary. We have to associate the
> data with a profile, which is then exposed as a configurable entity.
>
> Let's call them "connection profiles" to be very clear, since just
> "profile" can be ambiguous as well.

OK, I'll change next time I'll touch the code.

> MA> Well, currently a class is bound to a connection via a regexp, which is
> MA> used for checking the result of `file-remote-p'. This is good for Tramp,
> MA> but it is less good for url-* functionality unless `url-handler-mode' is
> MA> enabled, and it does not work for other applications like imap-based
> MA> connections.
>
> MA> Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
> MA> (or also a regexp?). All of them shall be optional.
>
> Wonderful. I propose that the application should be a defgroup symbol,
> and every connection profile should be required to have a defgroup.
>
> That would give us an implicit connection profile hierarchy and a place
> to hook connection profile customizations at the UI level.
>
> So defining connection profiles would be something like this:
>
> (add-connection-profiles :user U :defgroup X :protocol Y :machine Z
>                          :profile 'UXYZ :vars '(...))
>
> (add-connection-profiles :profile 'general :vars '(...))

Well, I'm a little bit lost. What does the defgroup here? I understand
that replaces what you have called initially an application, but I don't
see what defcustoms shall be placed now under the defgroup
hierarchy. Could you pls give an example?

I also know that there are users reluctant to the customization
machinery. Wouldn't we loose them?

> Thanks
> Ted

Best regards, Michael.

Re: bug#16026: Connection specific settings and proxies

[Ted Zlatanov]

On Sun, 12 Feb 2017 18:42:17 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> Your proposal looks OK to me. U, Y and Z shall be regexps, X a symbol
MA> (or also a regexp?). All of them shall be optional.
>> 
>> Wonderful. I propose that the application should be a defgroup symbol,
>> and every connection profile should be required to have a defgroup.
>> 
>> That would give us an implicit connection profile hierarchy and a place
>> to hook connection profile customizations at the UI level.
>> 
>> So defining connection profiles would be something like this:
>> 
>> (add-connection-profiles :user U :defgroup X :protocol Y :machine Z
>> :profile 'UXYZ :vars '(...))
>> 
>> (add-connection-profiles :profile 'general :vars '(...))

MA> Well, I'm a little bit lost. What does the defgroup here? I understand
MA> that replaces what you have called initially an application, but I don't
MA> see what defcustoms shall be placed now under the defgroup
MA> hierarchy. Could you pls give an example?

It's just a categorization. It would simply be the defgroup of an
existing application. That way you don't have to invent a new symbol for
every application that wants to use a connection profile and has the
defgroup declared already. For example, 'gnus for Gnus. So in `M-x
customize-group Gnus' you would maybe see an extra button to list and
modify the Gnus-specific connection profiles.

Connection profiles without a defgroup (application) symbol belong to
all applications.

MA> I also know that there are users reluctant to the customization
MA> machinery. Wouldn't we loose them?

Customize wouldn't be required. It's just an easy way to connect the
profile with other application-specific customizations, but users should
be able to add or modify profiles directly.

Does that make sense?

Ted

Re: bug#16026: Connection specific settings and proxies

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> Well, I'm a little bit lost. What does the defgroup here? I understand
> MA> that replaces what you have called initially an application, but I don't
> MA> see what defcustoms shall be placed now under the defgroup
> MA> hierarchy. Could you pls give an example?
>
> It's just a categorization. It would simply be the defgroup of an
> existing application. That way you don't have to invent a new symbol for
> every application that wants to use a connection profile and has the
> defgroup declared already. For example, 'gnus for Gnus. So in `M-x
> customize-group Gnus' you would maybe see an extra button to list and
> modify the Gnus-specific connection profiles.
>
> Connection profiles without a defgroup (application) symbol belong to
> all applications.

So we shall say that it's optional. Maybe you did already, and I've overseen.

> MA> I also know that there are users reluctant to the customization
> MA> machinery. Wouldn't we loose them?
>
> Customize wouldn't be required. It's just an easy way to connect the
> profile with other application-specific customizations, but users should
> be able to add or modify profiles directly.

We shouldn't even call this "defgroup", but still "application". We
might say then, if the application symbol is also an existing defgroup,
then there will be bonus features in the customization. But it still
shall work if a corresponding defgroup doesn't exist.

> Does that make sense?

I'm still sorting out. Will try to puzzle an example in order to see how
it goes. (And as usual, much to less time to spend)

> Ted

Best regards, Michael.

Re: bug#16026: Connection specific settings and proxies

[Ted Zlatanov]

On Mon, 13 Feb 2017 17:17:22 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> We shouldn't even call this "defgroup", but still "application". We
MA> might say then, if the application symbol is also an existing defgroup,
MA> then there will be bonus features in the customization. But it still
MA> shall work if a corresponding defgroup doesn't exist.

Yes, exactly.

MA> I'm still sorting out. Will try to puzzle an example in order to see how
MA> it goes. (And as usual, much to less time to spend)

OK, let me know. We can write some tests against the matcher when you're
ready to try it practically and to establish the API. Thanks for looking
at this, I am excited about it.

Ted

Re: Connection specific settings and proxies

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> I'm still sorting out. Will try to puzzle an example in order to see how
> MA> it goes. (And as usual, much to less time to spend)
>
> OK, let me know. We can write some tests against the matcher when you're
> ready to try it practically and to establish the API. Thanks for looking
> at this, I am excited about it.

I've pushed a change to master, which rewrites the connection-local
variables feature. `class' is now called `profile', and a criteria is
now a plist with property names :application, :protocol, :user and
:machine.

See (info "(elisp) Connection Local Variables")

I haven't added the interface to customization (yet); this will follow
next days.

If you need other/changed interfaces, let me know. Bug reports are also
welcome :-)

> Ted

Best regards, Michael.

bug#16026: Connection specific settings and proxies

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

>> So defining connection profiles would be something like this:
>>
>> (add-connection-profiles :user U :defgroup X :protocol Y :machine Z
>>                          :profile 'UXYZ :vars '(...))
>>
>> (add-connection-profiles :profile 'general :vars '(...))
>
> Well, I'm a little bit lost. What does the defgroup here? I understand
> that replaces what you have called initially an application, but I don't
> see what defcustoms shall be placed now under the defgroup
> hierarchy. Could you pls give an example?
>
> I also know that there are users reluctant to the customization
> machinery. Wouldn't we loose them?

Yeah, I think a special mode to tweak connection specific variables
would be better, perhaps...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Connection specific settings and proxies

[Daniel McClanahan]

On Wed, Feb 1, 2017 at 9:43 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> And, like I said, I don't know whether it's the right design choice to have
> these settings be global, or whether they should be passed in explicitly from
> each application.  Would users want to use one set of proxies while reading
> HTML news from Gnus and another when reading from eww?  Perhaps?  Perhaps not?

I'm no expert, but I've often been frustrated by configuration options that
aren't exposed as defcustoms. I think a defcustom of an alist mapping the match
domain to a list/plist of proxy settings would work. If `open-network-stream'
only accepts proxy settings as an argument and Gnus/eww/etc wants to allow
user-specified proxies, it seems each would have to implement its own defcustom
(or some other method) to handle that anyway; configuring these for each package
seems annoying. Callers can use a let binding if they want to override it, which
is about as easy as an argument.

A single global value would make it difficult to have different settings for
different callers. One potential solution is to have an optional "tag" symbol as
part of the proxy settings, which would only use the proxy if the caller
specified the same tag, maybe as a buffer-local variable. So Gnus could set the
tag to 'gnus or something in its mode, and a user could add a proxy entry with
the 'gnus tag. This might be confusing, though.

I don't use proxies too extensively, but something like that would have removed
the need for some hacky defadvice I used to configure my ssh connections on a
corporate network last summer. Just a thought.