expand tls to elpa.gnu.org

expand tls to elpa.gnu.org

[axel.junker]

Hey,

so far there is no tls/ssl support for elpa.gnu.org . In my opinion this is a real problem as there is no way to check the authenticity and integrity of downloaded packages. Is it possible to expand the certificate of gnu.org to elpa.gnu.org?
Of course this makes the package-manager not checking integrity - but I think anyone interested in doing so can modify it without problems.

Thanks,
/-Axel

Re: expand tls to elpa.gnu.org

[Ted Zlatanov]

On Sat, 19 Mar 2011 12:21:24 -0400 axel.junker@Safe-mail.net wrote: 

aj> so far there is no tls/ssl support for elpa.gnu.org . In my opinion
aj> this is a real problem as there is no way to check the authenticity
aj> and integrity of downloaded packages. Is it possible to expand the
aj> certificate of gnu.org to elpa.gnu.org?
aj> Of course this makes the package-manager not checking integrity - but I think anyone interested in doing so can modify it without problems.

I can install a certificate but it has to be requested by the domain
owner so I'm not sure who to bug about it.

Ted

Re: expand tls to elpa.gnu.org

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> aj> so far there is no tls/ssl support for elpa.gnu.org . In my opinion
> aj> this is a real problem as there is no way to check the authenticity
> aj> and integrity of downloaded packages. Is it possible to expand the
> aj> certificate of gnu.org to elpa.gnu.org?
> aj> Of course this makes the package-manager not checking integrity -
> aj> but I think anyone interested in doing so can modify it without
> aj> problems.
>
> I can install a certificate but it has to be requested by the domain
> owner so I'm not sure who to bug about it.

Why not simply distribute the certificate file with Emacs?

Also, the Emacs package manager uses the url library for downloading via
http.  How well does that library support https?  If I give
`url-retrieve-synchronously' a https url, does it currently DTRT?

url library and GnuTLS, and Emacs-issued certificates (was: expand tls to elpa.gnu.org)

[Ted Zlatanov]

On Mon, 21 Mar 2011 17:17:20 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> Ted Zlatanov <tzz@lifelogs.com> writes:
aj> so far there is no tls/ssl support for elpa.gnu.org . In my opinion
aj> this is a real problem as there is no way to check the authenticity
aj> and integrity of downloaded packages. Is it possible to expand the
aj> certificate of gnu.org to elpa.gnu.org?
aj> Of course this makes the package-manager not checking integrity -
aj> but I think anyone interested in doing so can modify it without
aj> problems.
>> 
>> I can install a certificate but it has to be requested by the domain
>> owner so I'm not sure who to bug about it.

CY> Why not simply distribute the certificate file with Emacs?

I assumed we'd want https://elpa.gnu.org/packages/ to look reasonable in
a web browser.

In any case, I think it's a good idea to set up an Emacs Certificate
Authority (CA) so we can create certificates that Emacs will trust.  We
just need to ship the CA's certificate with Emacs then, not every
certificate it has signed.  We can then make a .p12 file that browser
users can import to trust Emacs-signed certificates.

It may make sense, though, to make this CA a facility for the whole GNU
project and then the Emacs CA can be an intermediate CA hanging off that
root CA.  That should be decided before we start pushing out
certificates, please, so we don't have to invalidate them later.

CY> Also, the Emacs package manager uses the url library for downloading via
CY> http.  How well does that library support https?  If I give
CY> `url-retrieve-synchronously' a https url, does it currently DTRT?

It's insecure currently and won't work on all platforms.  It uses tls.el
(see `url-https-create-secure-wrapper') which in turn relies on the
gnutls-cli or openssl binaries to be installed and usable, calling
gnutls-cli by default with --insecure (though the user can manually
adjust that, see `tls-checktrust').  We need the GnuTLS support at the C
level to make the url library secure through gnutls.el.  

I need to look at Claudio Bley's patch that was posted on emacs-devel 2
days ago and figure out what's wrong with hostname verification against
the certificate.  Once that's done we can promote gnutls.el+gnutls.c to
"need testing" and make them the default for the url library, Gnus, etc.

If anyone wants to help with any part of this process, please let me
know.  I'm slow, especially at the C level, so Claudio's help was very
welcome.

Thanks
Ted

Re: url library and GnuTLS, and Emacs-issued certificates

[Ted Zlatanov]

On Mon, 21 Mar 2011 17:33:33 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Mon, 21 Mar 2011 17:17:20 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 
CY> Ted Zlatanov <tzz@lifelogs.com> writes:
aj> so far there is no tls/ssl support for elpa.gnu.org . In my opinion
aj> this is a real problem as there is no way to check the authenticity
aj> and integrity of downloaded packages. Is it possible to expand the
aj> certificate of gnu.org to elpa.gnu.org?
aj> Of course this makes the package-manager not checking integrity -
aj> but I think anyone interested in doing so can modify it without
aj> problems.
>>> 
>>> I can install a certificate but it has to be requested by the domain
>>> owner so I'm not sure who to bug about it.

CY> Why not simply distribute the certificate file with Emacs?

TZ> I assumed we'd want https://elpa.gnu.org/packages/ to look reasonable in
TZ> a web browser.

TZ> In any case, I think it's a good idea to set up an Emacs Certificate
TZ> Authority (CA) so we can create certificates that Emacs will trust.  We
TZ> just need to ship the CA's certificate with Emacs then, not every
TZ> certificate it has signed.  We can then make a .p12 file that browser
TZ> users can import to trust Emacs-signed certificates.

TZ> It may make sense, though, to make this CA a facility for the whole GNU
TZ> project and then the Emacs CA can be an intermediate CA hanging off that
TZ> root CA.  That should be decided before we start pushing out
TZ> certificates, please, so we don't have to invalidate them later.

Any opinions on this?  It's really not hard to set up the CA stuff but
I'd like to know what people think before I do it.  It really seems like
it should be a GNU-level or FSF-level facility.

CY> Also, the Emacs package manager uses the url library for downloading via
CY> http.  How well does that library support https?  If I give
CY> `url-retrieve-synchronously' a https url, does it currently DTRT?

TZ> It's insecure currently and won't work on all platforms.  It uses tls.el
TZ> (see `url-https-create-secure-wrapper') which in turn relies on the
TZ> gnutls-cli or openssl binaries to be installed and usable, calling
TZ> gnutls-cli by default with --insecure (though the user can manually
TZ> adjust that, see `tls-checktrust').  We need the GnuTLS support at the C
TZ> level to make the url library secure through gnutls.el.  

TZ> I need to look at Claudio Bley's patch that was posted on emacs-devel 2
TZ> days ago and figure out what's wrong with hostname verification against
TZ> the certificate.  Once that's done we can promote gnutls.el+gnutls.c to
TZ> "need testing" and make them the default for the url library, Gnus, etc.

This work is almost done.  But probably a better approach than relying
directly on gnutls.el is to make url.el use proto-stream.el from Gnus,
which handles most of the connection details automatically whether Emacs
has GnuTLS support build-in or not.  I looked at it in order to make the
new GnuTLS support work properly and it seems like a good general
facility, not just for Gnus.

proto-stream.el doesn't depend on any Gnus internals, it's a standalone
library.  It could live in net/ in the Emacs repo.

Thanks
Ted

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> TZ> In any case, I think it's a good idea to set up an Emacs
> TZ> Certificate Authority (CA) so we can create certificates that
> TZ> Emacs will trust...  It may make sense, though, to make this CA a
> TZ> facility for the whole GNU project and then the Emacs CA can be an
> TZ> intermediate CA hanging off that root CA.  That should be decided
> TZ> before we start pushing out certificates, please, so we don't have
> TZ> to invalidate them later.
>
> Any opinions on this?  It's really not hard to set up the CA stuff but
> I'd like to know what people think before I do it.  It really seems
> like it should be a GNU-level or FSF-level facility.

I don't think setting up a GNU-wide CA is a good idea; it's mission
creep and the gains seem negligible.  As for an Emacs-specific CA, I
don't know enough of the details of how CAs are maintained to evaluate
the proposal.

On reflection, the best solution is the one that needs the least work
from us.  So it's probably best to ask the FSF sysadmins to request and
install a cert, as you originally suggested.  Could you email them?

> This work is almost done.  But probably a better approach than relying
> directly on gnutls.el is to make url.el use proto-stream.el from Gnus,
> which handles most of the connection details automatically whether Emacs
> has GnuTLS support build-in or not.  I looked at it in order to make the
> new GnuTLS support work properly and it seems like a good general
> facility, not just for Gnus.
>
> proto-stream.el doesn't depend on any Gnus internals, it's a standalone
> library.  It could live in net/ in the Emacs repo.

How bout merging the open-protocol-stream code directly into
open-network-stream?  Then we can make open-protocol-stream an alias for
open-network-stream, and (provide 'proto-stream) in subr.el.

If the Gnus developers don't object, I propose to do this.

(Also, gnutls.el should be changed to explicitly recommend that
applications not use it directly, and we should merge net/tls.el and
gnus/starttls.el; those two packages appear to be duplicates.)

Re: url library and GnuTLS, and Emacs-issued certificates

[Ted Zlatanov]

On Wed, 23 Mar 2011 14:31:02 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

>> This work is almost done.  But probably a better approach than relying
>> directly on gnutls.el is to make url.el use proto-stream.el from Gnus,
>> which handles most of the connection details automatically whether Emacs
>> has GnuTLS support build-in or not.  I looked at it in order to make the
>> new GnuTLS support work properly and it seems like a good general
>> facility, not just for Gnus.
>> 
>> proto-stream.el doesn't depend on any Gnus internals, it's a standalone
>> library.  It could live in net/ in the Emacs repo.

CY> How bout merging the open-protocol-stream code directly into
CY> open-network-stream?  Then we can make open-protocol-stream an alias for
CY> open-network-stream, and (provide 'proto-stream) in subr.el.

CY> If the Gnus developers don't object, I propose to do this.

OK with me, as long as it doesn't break Gnus.  I'd like to get Lars'
opinion too.

CY> (Also, gnutls.el should be changed to explicitly recommend that
CY> applications not use it directly, 

How about this change (adding the second sentence):

;;; Commentary:

;; This package provides language bindings for the GnuTLS library
;; using the corresponding core functions in gnutls.c.  It should NOT
;; be used directly, only through open-protocol-stream.

CY> and we should merge net/tls.el and gnus/starttls.el; those two
CY> packages appear to be duplicates.)

Agreed, as long as Lars is OK with this.

Ted

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> How bout merging the open-protocol-stream code directly into
> open-network-stream?  Then we can make open-protocol-stream an alias for
> open-network-stream, and (provide 'proto-stream) in subr.el.
>
> If the Gnus developers don't object, I propose to do this.

It sounds like a workable idea to me.  Since `open-protocol-stream' is a
superset of `open-network-stream', really, it should hopefully not break
other third-party usages of `open-network-stream', but I think we should
have a closer look at the `open-protocol-stream' logic to make sure
first.

But my immediate reaction is that this would be a good idea.

> (Also, gnutls.el should be changed to explicitly recommend that
> applications not use it directly, and we should merge net/tls.el and
> gnus/starttls.el; those two packages appear to be duplicates.)

Well, they aren't duplicates per se.  tls.el does straight up TLS, while
starttls.el does STARTTLS.  The way starttls.el interfaces with the
external programs is pretty hairy (doing some unencrypted traffic, then
sending a signal to the program, and then doing encrypted traffic), and
probably deserves its own elisp file, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> we should merge net/tls.el and gnus/starttls.el; those two packages
>> appear to be duplicates.
>
> Well, they aren't duplicates per se.  tls.el does straight up TLS, while
> starttls.el does STARTTLS.  The way starttls.el interfaces with the
> external programs is pretty hairy (doing some unencrypted traffic, then
> sending a signal to the program, and then doing encrypted traffic), and
> probably deserves its own elisp file, I think.

Could you explain in more detail?  What I see is this:

;;; tls.el --- TLS/SSL support via wrapper around GnuTLS
...
;; This package implements a simple wrapper around "gnutls-cli" to
;; make Emacs support TLS/SSL.
...
(defun open-tls-stream (name buffer host port)
  "Open a TLS connection for a port to a host.
Returns a subprocess-object to represent the connection.
..


;;; starttls.el --- STARTTLS functions
...
;; This file now contains a combination of the two previous
;; implementations both called "starttls.el".  The first one is Daiki
;; Ueno's starttls.el which uses his own "starttls" command line tool,
;; and the second one is Simon Josefsson's starttls.el which uses
;; "gnutls-cli" from GNUTLS.
...
(defun starttls-open-stream (name buffer host port)
  "Open a TLS connection for a port to a host.
Returns a subprocess object to represent the connection.
...


Both use the gnutls-cli command tool (though starttls.el apparently
supports starttls too), and both implement a wrapper around
open-network-stream.  "Unnecessary duplication" is surely the natural
reaction...

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> Could you explain in more detail?

tls.el is quite simple.  You start openssl/gnutls-cli as a subprocess
with the correct parameters, parse the output slightly, and then talk
over the socket.

starttls.el is a bit different.  You call `open-starttls-stream', which
starts gnutls-cli in unencrypted mode.  Then you do protocol-specific
things to see whether the server supports STARTTLS.  If it does, you
issue the commands necessary for the server to start talking TLS.  You
then call `startls-negotiate' which sends a signal to gnutls-cli which
makes gnutls-cli go to encrypted mode, parses the output from
gnutls-cli, and returns the status of the now-encrypted socket.

To make things even more confusing, openssl does have support for
STARTTLS, too, but you then have to let openssl itself do the STARTTLS
negotiation by telling openssl what protocol it's talking.  And it
doesn't support all the protocols that Emacs needs, so it can't really
be used.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> tls.el is quite simple.  You start openssl/gnutls-cli as a subprocess
> with the correct parameters, parse the output slightly, and then talk
> over the socket.
>
> starttls.el is a bit different.  You call `open-starttls-stream', which
> starts gnutls-cli in unencrypted mode.  Then you do protocol-specific
> things to see whether the server supports STARTTLS.  If it does, you
> issue the commands necessary for the server to start talking TLS.  You
> then call `startls-negotiate' which sends a signal to gnutls-cli which
> makes gnutls-cli go to encrypted mode, parses the output from
> gnutls-cli, and returns the status of the now-encrypted socket.

I see.  Thanks for the explanation.  A couple more questions:

How about gnutls.el?  If I understand correctly, open-gnutls-stream
works just like open-tls-stream, except it uses the gnutls library
directly instead of using a command line client---and it does not
support the mode of operation provided in starttls.el.  Correct?

Also, does this mean it's impossible to use open-tls-stream and
open-gnutls-stream to opportunistically open a TLS stream?  I see that
proto-stream.el, in `network' mode, opens a connection using
starttls-open-stream and then uses gnutls-negotiate from gnutls.el to do
the negotiation; what's the advantage of doing this?

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> How bout merging the open-protocol-stream code directly into
>> open-network-stream?  Then we can make open-protocol-stream an alias for
>> open-network-stream, and (provide 'proto-stream) in subr.el.
>
> It sounds like a workable idea to me.  Since `open-protocol-stream' is a
> superset of `open-network-stream', really, it should hopefully not break
> other third-party usages of `open-network-stream', but I think we should
> have a closer look at the `open-protocol-stream' logic to make sure
> first.
>
> But my immediate reaction is that this would be a good idea.

That's great.

There is one main snag: open-protocol-stream returns a four-element
list, (PROC GREETING CAPABILITIES TYPE), while open-network-stream
returns just PROC.

Within Gnus, the call to open-protocol-stream in nntp-open-connection
discards the other list elements anyway; but nnimap-open-connection-1
does use them.

How about if we change this by adding a :return-list parameter that
makes open-network-stream (merged from open-protocol-stream) return a
list rather than a process object?  Then, in the absence of this
parameter, only a process object would be returned.

Re: url library and GnuTLS, and Emacs-issued certificates

[Ted Zlatanov]

On Thu, 24 Mar 2011 15:23:15 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> There is one main snag: open-protocol-stream returns a four-element
CY> list, (PROC GREETING CAPABILITIES TYPE), while open-network-stream
CY> returns just PROC.

CY> Within Gnus, the call to open-protocol-stream in nntp-open-connection
CY> discards the other list elements anyway; but nnimap-open-connection-1
CY> does use them.

CY> How about if we change this by adding a :return-list parameter that
CY> makes open-network-stream (merged from open-protocol-stream) return a
CY> list rather than a process object?  Then, in the absence of this
CY> parameter, only a process object would be returned.

Can the return list be made, instead, into a property list after the
first element?  Something like

(PROC :greeting GREETING :capabilities CAPABILITIES :type TYPE)

Since only Gnus uses the extended format it should be OK to do this;
it's backwards compatible and can be extended further.

Ted

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> How about if we change this by adding a :return-list parameter that
> makes open-network-stream (merged from open-protocol-stream) return a
> list rather than a process object?  Then, in the absence of this
> parameter, only a process object would be returned.

Sounds good.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> How about gnutls.el?  If I understand correctly, open-gnutls-stream
> works just like open-tls-stream, except it uses the gnutls library
> directly instead of using a command line client---and it does not
> support the mode of operation provided in starttls.el.  Correct?

No, with gnutls.c, we just open a normal network stream (with
`open-network-stream'), and then if it turns out that the server
supports STARTTLS, we just put the server in STARTTLS mode and then
issue a `gnutls-negotiate' call.

> Also, does this mean it's impossible to use open-tls-stream and
> open-gnutls-stream to opportunistically open a TLS stream?  I see that
> proto-stream.el, in `network' mode, opens a connection using
> starttls-open-stream and then uses gnutls-negotiate from gnutls.el to do
> the negotiation; what's the advantage of doing this?

Looking at the code in `proto-stream-open-network', I can see why you
think it's doing that, but it's not.  I hope.  :-)  That function is
somewhat hairy because of all the combinations in the
do-opportunistic-upgrade/do-forced-upgrade/starttls.el/gnutls.c/do-support-STARTTLS
matrix, as well as error handling...

But for the "gnutls where the server supports STARTTLS and there's no
errors" case, it should just be doing

(defun proto-stream-open-network (name buffer host service parameters)

[...]

	 (stream (open-network-stream name buffer host service))

[...]

	    ;; The server said it was OK to start doing STARTTLS negotiations.
	    (if (fboundp 'open-gnutls-stream)
		(gnutls-negotiate stream nil)                

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

On Wed, 23 Mar 2011 14:31:02 -0400 Chong Yidong <cyd@stupidchicken.com> wrote: 

CY> On reflection, the best solution is the one that needs the least work
CY> from us.  So it's probably best to ask the FSF sysadmins to request and
CY> install a cert, as you originally suggested.  Could you email them?

To support HTTPS URLs, we'll at least need the appended patch (so
https:// patterns are recognized).  I also explicitly mention that file:
URLs should be directory names even though it's implicit otherwise
because I guarantee you, people will try it anyhow.  See what you think.

The certificate is installed and https://elpa.gnu.org/ should be up as
soon as the firewall is opened for port 443.  When the GnuTLS patch is
in the trunk we can modify url-http.el as needed to use GnuTLS.

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: package-https.patch --]
[-- Type: text/x-diff, Size: 1170 bytes --]

=== modified file 'lisp/emacs-lisp/package.el'
--- lisp/emacs-lisp/package.el	2011-03-19 18:27:55 +0000
+++ lisp/emacs-lisp/package.el	2011-03-26 11:36:47 +0000
@@ -224,8 +224,9 @@
 Each element has the form (ID . LOCATION).
  ID is an archive name, as a string.
  LOCATION specifies the base location for the archive.
-  If it starts with \"http:\", it is treated as a HTTP URL;
+  If it starts with \"http:\" or \"https:\", it is treated as a HTTP URL;
   otherwise it should be an absolute directory name.
+  (file: URLs should simply be entered as directory names!)
   (Other types of URL are currently not supported.)"
   :type '(alist :key-type (string :tag "Archive name")
                 :value-type (string :tag "URL or directory name"))
@@ -652,7 +653,7 @@
 This macro retrieves FILE from LOCATION into a temporary buffer,
 and evaluates BODY while that buffer is current.  This work
 buffer is killed afterwards.  Return the last value in BODY."
-  `(let* ((http (string-match "\\`http:" ,location))
+  `(let* ((http (string-match "\\`\\(https\\|http\\):" ,location))
 	  (buffer
 	   (if http
 	       (url-retrieve-synchronously (concat ,location ,file))

Re: url library and GnuTLS, and Emacs-issued certificates

[Reiner Steib]

On Sat, Mar 26 2011, Ted Zlatanov wrote:

> -  `(let* ((http (string-match "\\`http:" ,location))
> +  `(let* ((http (string-match "\\`\\(https\\|http\\):" ,location))

-  `(let* ((http (string-match "\\`http:" ,location))
+  `(let* ((http (string-match "\\`https?:" ,location))

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: url library and GnuTLS, and Emacs-issued certificates

[Ted Zlatanov]

On Sat, 26 Mar 2011 14:39:51 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Sat, Mar 26 2011, Ted Zlatanov wrote:
>> -  `(let* ((http (string-match "\\`http:" ,location))
>> +  `(let* ((http (string-match "\\`\\(https\\|http\\):" ,location))

RS> -  `(let* ((http (string-match "\\`http:" ,location))
RS> +  `(let* ((http (string-match "\\`https?:" ,location))

(I *knew* I'd have to explain it :)

It's rarely used so I wasn't worried about optimizing it.  Showing both
versions in an alternation makes it easier to see both schemes are
supported at a glance.

Ted

Re: url library and GnuTLS, and Emacs-issued certificates

[Tom Tromey]

>>>>> "Ted" == Ted Zlatanov <tzz@lifelogs.com> writes:

Ted> To support HTTPS URLs, we'll at least need the appended patch (so
Ted> https:// patterns are recognized).

This business about (re-search-forward "^$") was just a workaround for a
url oddity -- HTTP puts some headers in the buffer, but other protocols
may not.  It would be cleaner to fix this in the URL package itself.
Then a special case for HTTP would not be needed.

Ted> I also explicitly mention that file: URLs should be directory names
Ted> even though it's implicit otherwise because I guarantee you, people
Ted> will try it anyhow.

I don't deeply care, but I think if something is documented as using
URLs, then it is fine to just require them.

Tom

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> How about if we change this by adding a :return-list parameter that
>> makes open-network-stream (merged from open-protocol-stream) return a
>> list rather than a process object?  Then, in the absence of this
>> parameter, only a process object would be returned.
>
> Sounds good.

OK, I have changed open-protocol-stream accordingly.  I have also made a
few more changes:

 - TZZ's suggestion to make the list return value (PROC . PLIST)
   for extensibility.

 - Instead of returning nil on connection errors, it returns the killed
   process, like open-network-stream.

 - Renamed `network' to the more explicit `try-starttls'.
   Renamed `network-only' to `default'.

 - Changed the default connection type to `default' (no encryption), so
   that open-protocol-stream with no arguments will be equivalent to
   open-network-stream.

I have changed the Gnus callers to accomodate these changes, including
making the connection type `try-starttls' by default.  NNTP and NNIMAP
seem to work the same as before, but let me know if anything breaks for
you.

The next step is to merge proto-steam.el into open-network-stream in the
Emacs trunk.  The present revisison of proto-stream.el (103758) can
probably be used for the stand-alone Gnus.

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Tom Tromey <tromey@redhat.com> writes:

> Ted> I also explicitly mention that file: URLs should be directory names
> Ted> even though it's implicit otherwise because I guarantee you, people
> Ted> will try it anyhow.
>
> I don't deeply care, but I think if something is documented as using
> URLs, then it is fine to just require them.

It's better not to allow file:// URLs here, because then we have to deal
with URL quoting for filenames.

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

>  - Renamed `network' to the more explicit `try-starttls'.
>    Renamed `network-only' to `default'.

So...  there is no `network' any more?

> I have changed the Gnus callers to accomodate these changes, including
> making the connection type `try-starttls' by default.  NNTP and NNIMAP
> seem to work the same as before, but let me know if anything breaks for
> you.

People have select methods with `network' specified already, so this
will break Gnus for them pretty badly.

Which is why I made `network' be the opportunistic-STARTTLS-upgrade
name.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

>  - Changed the default connection type to `default' (no encryption), so
>    that open-protocol-stream with no arguments will be equivalent to
>    open-network-stream.

But it already is.  If you don't supply a `capability-command'
parameter, it doesn't try to do any upgrades.

So I can't really see that your renaming of these values does anything
useful.  But it does break existing Gnus users pretty badly.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> I have changed the Gnus callers to accomodate these changes, including
>> making the connection type `try-starttls' by default.  NNTP and NNIMAP
>> seem to work the same as before, but let me know if anything breaks for
>> you.
>
> People have select methods with `network' specified already, so this
> will break Gnus for them pretty badly.
>
> Which is why I made `network' be the opportunistic-STARTTLS-upgrade
> name.

Isn't proto-stream.el and the `network'/`network-only' keywords new in
this version of Gnus?

We could, of course, add `network' as a synonym for `try-starttls' and
`network-only' as a synonym.

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>>  - Changed the default connection type to `default' (no encryption), so
>>    that open-protocol-stream with no arguments will be equivalent to
>>    open-network-stream.
>
> But it already is.  If you don't supply a `capability-command'
> parameter, it doesn't try to do any upgrades.

That is poor semantics---it means that "type" is not really determined
by :type, but by some combination of :type and the other arguments that
may or may not be present.

> But it does break existing Gnus users pretty badly.

In what way?  Unless I missed something, the existing callers to
open-protocol-stream now provide :type values that give exactly the same
behavior as before.

Re: url library and GnuTLS, and Emacs-issued certificates

[Tom Tromey]

>>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:

Chong> Tom Tromey <tromey@redhat.com> writes:
Ted> I also explicitly mention that file: URLs should be directory names
Ted> even though it's implicit otherwise because I guarantee you, people
Ted> will try it anyhow.

Tom> I don't deeply care, but I think if something is documented as using
Tom> URLs, then it is fine to just require them.

Chong> It's better not to allow file:// URLs here, because then we have to deal
Chong> with URL quoting for filenames.

I don't understand.  Does the URL package not handle this?

Tom

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Tom Tromey <tromey@redhat.com> writes:

> Ted> I also explicitly mention that file: URLs should be directory names
> Ted> even though it's implicit otherwise because I guarantee you, people
> Ted> will try it anyhow.
>
> Tom> I don't deeply care, but I think if something is documented as
> Tom> using URLs, then it is fine to just require them.
>
> Chong> It's better not to allow file:// URLs here, because then we
> Chong> have to deal with URL quoting for filenames.
>
> I don't understand.  Does the URL package not handle this?

I assume what you're proposing is to require file:// URLs, instead of
ordinary file names, for the locations in `package-archives'.

The problem with that is that everywhere else in Emacs where we let
users specify file names, we use "ordinary" Emacs file names.  It would
be inconvenient for users to require them to remember how to do URL
quoting if they want to specify a local directory for this particular
variable.

If what you meant, instead, was that we should support file:// URLs in
*addition* to ordinary file names, I guess there is no harm doing that,
as long as we allow ordinary file names too.

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> Isn't proto-stream.el and the `network'/`network-only' keywords new in
> this version of Gnus?

`network-only' is new.  People have been using `network' as the stream
spec for nntp/nnimap since forever.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

>> But it already is.  If you don't supply a `capability-command'
>> parameter, it doesn't try to do any upgrades.
>
> That is poor semantics---it means that "type" is not really determined
> by :type, but by some combination of :type and the other arguments that
> may or may not be present.

If you just feed `open-protocol-stream' `network' without a capability
command, you get a non-upgraded connection.  If you do the same with
`try-starttls' you will also not get a non-upgraded connection.

I don't really see why the latter is clearer than the former.

(But in the latter case you could error out if you do that and not
supply a `capability-command', I guess.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> > Isn't proto-stream.el and the `network'/`network-only' keywords new in
> > this version of Gnus?
>
> `network-only' is new.  People have been using `network' as the stream
> spec for nntp/nnimap since forever.

nntp doesn't look like a issue, since nntp-open-connection-function is
passed through an internal association list to get the connection type.

As for nnimap, fair enough.  This can be solved by adding `network' as a
synonym to `try-starttls'.

> If you just feed `open-protocol-stream' `network' without a capability
> command, you get a non-upgraded connection.  If you do the same with
> `try-starttls' you will also not get a non-upgraded connection.
>
> I don't really see why the latter is clearer than the former.

Explaining the meanings of the :type makes the former scheme sound more
complicated: "try starttls if the :capability-command and :success and
:starttls-function parameters are present, otherwise open a normal
network stream".  As opposed to "try starttls".

But if you feel strongly about this, I won't insist; it's not an
important issue.  Should I bring the default back?

> (But in the latter case you could error out if you do that and not
> supply a `capability-command', I guess.)

It would be cleaner to error out, IMO.  The :success and
:starttls-function parameters ought be checked also.

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

> Explaining the meanings of the :type makes the former scheme sound more
> complicated: "try starttls if the :capability-command and :success and
> :starttls-function parameters are present, otherwise open a normal
> network stream".  As opposed to "try starttls".
>
> But if you feel strongly about this, I won't insist; it's not an
> important issue.  Should I bring the default back?

To be concrete, the doc would say something like this:

  If :type is omitted or nil, that specifies a `try-starttls' type
  connection if the :success, :capability-command, and
  :starttls-function parameters are all also given; otherwise, a plain
  network connection.

Is that fine by you?

Re: url library and GnuTLS, and Emacs-issued certificates

[Reiner Steib]

On Sat, Mar 26 2011, Ted Zlatanov wrote:

> On Sat, 26 Mar 2011 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 
> RS> On Sat, Mar 26 2011, Ted Zlatanov wrote:
>>> -  `(let* ((http (string-match "\\`http:" ,location))
>>> +  `(let* ((http (string-match "\\`\\(https\\|http\\):" ,location))
>
> RS> -  `(let* ((http (string-match "\\`http:" ,location))
> RS> +  `(let* ((http (string-match "\\`https?:" ,location))
>
> (I *knew* I'd have to explain it :)
>
> It's rarely used so I wasn't worried about optimizing it.  

I didn't think about optimizing neither.  Though regexp-opt suggests
that "https?" is preferable:

    ELISP> (regexp-opt '("http" "https"))
    "https?"

> Showing both versions in an alternation makes it easier to see both
> schemes are supported at a glance.

I think "https?" also makes this clear and it is more readable -
especially since it doesn't require unnecessary grouping.

BTW:

emacs/lisp$ grep -e 'http[\\].*https' -e 'https[\\].*http' *.el */*.el|wc -l
1
emacs/lisp$ grep -e 'https[?]' *.el */*.el|wc -l
12

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: url library and GnuTLS, and Emacs-issued certificates

[Ted Zlatanov]

On Sat, 26 Mar 2011 07:07:02 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> The certificate is installed and https://elpa.gnu.org/ should be up as
TZ> soon as the firewall is opened for port 443.  When the GnuTLS patch is
TZ> in the trunk we can modify url-http.el as needed to use GnuTLS.

https://elpa.gnu.org/packages/ is now available and works exactly like
http://elpa.gnu.org/packages/ (it's served from the same document root).

Ted

Re: url library and GnuTLS, and Emacs-issued certificates

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> To be concrete, the doc would say something like this:
>
>   If :type is omitted or nil, that specifies a `try-starttls' type
>   connection if the :success, :capability-command, and
>   :starttls-function parameters are all also given; otherwise, a plain
>   network connection.
>
> Is that fine by you?

My point is that `try-starttls' is somewhat meaningless, since you have
to give all the additional parameters to get a STARTTLS upgrade.  If you
don't, you get a non-upgraded connection even if you do say
`try-starttls'.

That, and the nnimap incompatibility issue (sorry for not looking at the
nntp.el code first) just makes me feel that `try-starttls' isn't the
most useful choice.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url library and GnuTLS, and Emacs-issued certificates

[Chong Yidong]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> My point is that `try-starttls' is somewhat meaningless, since you have
> to give all the additional parameters to get a STARTTLS upgrade.  If you
> don't, you get a non-upgraded connection even if you do say
> `try-starttls'.
>
> That, and the nnimap incompatibility issue (sorry for not looking at the
> nntp.el code first) just makes me feel that `try-starttls' isn't the
> most useful choice.

OK, I shan't argue the point.  I'll commit the change bringing the
default back to `network', and then proceed to merge proto-stream.el
into open-network-stream.