serving ELPA over HTTP/S

serving ELPA over HTTP/S

[Ted Zlatanov]

What would it take to switch the GNU ELPA to HTTP/S?  It's already
accessible that way at https://elpa.gnu.org and all the web pages are
consistent.  I can think of the following:

* set the defaults and docs to point to https://elpa.gnu.org

admin/notes/elpa:the content visible at http://elpa.gnu.org/packages.
doc/lispref/package.texi:the archive hosted by the GNU project at @url{http://elpa.gnu.org}.  This
etc/NEWS.24:from a package repository at http://elpa.gnu.org.
etc/ORG-NEWS:available through http://elpa.gnu.org) and the =org-plus*.tar= package (the
lisp/emacs-lisp/package.el:(defcustom package-archives '(("gnu" . "http://elpa.gnu.org/packages/"))
doc/emacs/trouble.texi:or in Gnu ELPA (@url{https://elpa.gnu.org/}).

* warn and possibly abort when ELPA transfers are done over HTTP
* offer to switch the "gnu" ELPA archive to https://elpa.gnu.org
* test on all platforms
* maybe add the GNU ELPA SSL certificate chain explicitly to Emacs

Ted

Re: serving ELPA over HTTP/S

[Stefan Monnier]

> What would it take to switch the GNU ELPA to HTTP/S?

Nothing.  As you've seen it already just works.
If you mean "what would it take to make Emacs use the "https" protocol
by default?" then the answer is:

not very much.  You just need to make sure it still works when the
running Emacs does not support TLS.


        Stefan

Re: serving ELPA over HTTP/S

[Artur Malabarba]

> What would it take to switch the GNU ELPA to HTTP/S?

I don't know if other measures are necessary, but in the very least:

* Change `package-archives' initial value to '(("gnu" .
"https://elpa.gnu.org/packages/"))

Re: serving ELPA over HTTP/S

[Ted Zlatanov]

On Mon, 04 May 2015 10:22:06 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> What would it take to switch the GNU ELPA to HTTP/S?
SM> Nothing.  As you've seen it already just works.

Sorry, I meant "switch users to use the GNU ELA over HTTP/S."

I listed several items; do you think none of those are required?

SM> If you mean "what would it take to make Emacs use the "https" protocol
SM> by default?" then the answer is:

SM> not very much.  You just need to make sure it still works when the
SM> running Emacs does not support TLS.

Define "it works."

We can switch to an external binary for the data transfer, for instance.
But is that better than asking the user to enable the GnuTLS integration?
On what supported platforms is it simply not possible?  I don't know.
Are those platforms worth exposing our users to the drawbacks of
installing packages over HTTP?

Thanks
Ted

Re: serving ELPA over HTTP/S

[Stefan Monnier]

> I listed several items; do you think none of those are required?

Let's see:

* set the defaults and docs to point to https://elpa.gnu.org

Yes, that's the crux of the matter, and the only part I discussed in my
previous message.

* warn and possibly abort when ELPA transfers are done over HTTP

Why?

* offer to switch the "gnu" ELPA archive to https://elpa.gnu.org

Why?

* test on all platforms

Tests are good, but in most cases we rely on early users to test.

* maybe add the GNU ELPA SSL certificate chain explicitly to Emacs

IIUC it's not necessary because that should already be installed on
your system.

SM> not very much.  You just need to make sure it still works when the
SM> running Emacs does not support TLS.
> Define "it works."

You can still install packages via package.el.

> We can switch to an external binary for the data transfer, for instance.

Why bother?

> But is that better than asking the user to enable the GnuTLS integration?

I don't think so.

> On what supported platforms is it simply not possible?  I don't know.

I don't think it matters.

> Are those platforms worth exposing our users to the drawbacks of
> installing packages over HTTP?

I don't think those drawbacks are so terrible.
But, yes, by all means, do try and change package-archives to default to
using https when that works.


        Stefan

Re: serving ELPA over HTTP/S

[Ted Zlatanov]

On Mon, 04 May 2015 11:41:09 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I listed several items; do you think none of those are required?
SM> Let's see:

SM> * set the defaults and docs to point to https://elpa.gnu.org

SM> Yes, that's the crux of the matter, and the only part I discussed in my
SM> previous message.

SM> * warn and possibly abort when ELPA transfers are done over HTTP
SM> * offer to switch the "gnu" ELPA archive to https://elpa.gnu.org

SM> Why?

Because HTTP is worse than HTTP/S as a software delivery channel in
almost every way.

SM> * maybe add the GNU ELPA SSL certificate chain explicitly to Emacs

SM> IIUC it's not necessary because that should already be installed on
SM> your system.

Unfortunately it's not always the case. If we want package installation
to always work, we should provide a safety net.  But I'm not insisting
on this, only suggesting it.

>> We can switch to an external binary for the data transfer, for instance.

SM> Why bother?

To provide a fallback.  But I agree that it's better to just ask for GnuTLS.

>> Are those platforms worth exposing our users to the drawbacks of
>> installing packages over HTTP?

SM> I don't think those drawbacks are so terrible.

OK, let's say they are enough to justify some concern.  So we should
express that concern through warnings.

SM> But, yes, by all means, do try and change package-archives to default to
SM> using https when that works.

OK. Perhaps it's best to simply make it a list instead of a string and
try each one in sequence. That would also allow `package-archives' to
support "dumb" failover natively, a nice side benefit!

The change will be from:

    LOCATION specifies the base location for the archive.
     If it starts with "http:", it is treated as a HTTP URL;
     otherwise it should be an absolute directory name.
     (Other types of URL are currently not supported.)

to:

    LOCATION specifies the base location for the archive.
     If it starts with "http:" or "https:", it is treated as a HTTP URL;
     otherwise it should be an absolute directory name.
     (Other types of URL are currently not supported.)
     If LOCATION is a list, each element is attempted as a URL sequentially.

And change the default to

    (("gnu" "https://elpa.gnu.org/packages/" "http://elpa.gnu.org/packages/"))

That would annoy but not break users that don't have GnuTLS available.

Sounds good?

Ted

Re: serving ELPA over HTTP/S

[Ivan Shmakov]

>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:
>>>>> On Mon, 04 May 2015 11:41:09 -0400 Stefan Monnier wrote:

[…]

 TZ> * warn and possibly abort when ELPA transfers are done over HTTP *
 TZ> offer to switch the "gnu" ELPA archive to https://elpa.gnu.org

 SM> Why?

 TZ> Because HTTP is worse than HTTP/S as a software delivery channel in
 TZ> almost every way.

	… Except for the cache-ability, where HTTP is a sure winner.

	Debian, which is known to use HTTP for deliver .deb packages for
	decades now, even offers specialized software to use as an HTTP
	caching proxy for APT repositories – such as apt-cacher,
	apt-cacher-ng and approx.  The software which isn’t entirely out
	of any demand; consider, e. g., [1]:

 rank name                            inst  vote   old recent no-files
2036  apt-cacher-ng                   1473   964   411    97     1
3408  apt-cacher                       358   297    51     8     2
3677  approx                           288   246    39     3     0

[1] http://popcon.debian.org/source/by_vote.gz

[…]

-- 
FSF associate member #7257  http://am-1.org/~ivan/      … 3013 B6A0 230E 334A

Re: serving ELPA over HTTP/S

[Stefan Monnier]

SM> * warn and possibly abort when ELPA transfers are done over HTTP
SM> * offer to switch the "gnu" ELPA archive to https://elpa.gnu.org
SM> Why?
> Because HTTP is worse than HTTP/S as a software delivery channel in
> almost every way.

Better/worse is not sufficient in itself to justify annoying the user.
So the "why" was really saying "why is it a big enough deal"?

SM> * maybe add the GNU ELPA SSL certificate chain explicitly to Emacs
SM> IIUC it's not necessary because that should already be installed on
SM> your system.
> Unfortunately it's not always the case. If we want package installation
> to always work, we should provide a safety net.  But I'm not insisting
> on this, only suggesting it.

Package installation should indeed work even without those
certificate chains.  Either by accepting the "unverified" certificates,
or by falling back to HTTP.

>>> We can switch to an external binary for the data transfer, for instance.
SM> Why bother?
> To provide a fallback.  But I agree that it's better to just ask for GnuTLS.

We already have a fallback to HTTP.  I think it's sufficient.

> OK. Perhaps it's best to simply make it a list instead of a string and
> try each one in sequence.

Why?  Why not just

  (if (we-have-gnutls) <thehttpsurl> <thehttpurl>)


-- Stefan

Re: serving ELPA over HTTP/S

[Ted Zlatanov]

On Mon, 04 May 2015 15:16:01 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> * warn and possibly abort when ELPA transfers are done over HTTP
SM> * offer to switch the "gnu" ELPA archive to https://elpa.gnu.org
SM> Why?
>> Because HTTP is worse than HTTP/S as a software delivery channel in
>> almost every way.

SM> Better/worse is not sufficient in itself to justify annoying the user.
SM> So the "why" was really saying "why is it a big enough deal"?

Clearly, because I think it's worth annoying the user, since plain HTTP
should be the exception.

SM> Package installation should indeed work even without those
SM> certificate chains.  Either by accepting the "unverified" certificates,
SM> or by falling back to HTTP.

OK, I think we can accommodate that.

>> OK. Perhaps it's best to simply make it a list instead of a string and
>> try each one in sequence.

SM> Why?  Why not just

SM>   (if (we-have-gnutls) <thehttpsurl> <thehttpurl>)

1) so ELPA archives can have multiple URLs. Assuming there's just one is
not ideal in the long term.

2) because I don't think this should be set in the code.  It's a user choice.

On Mon, 04 May 2015 17:20:46 +0000 Ivan Shmakov <ivan@siamics.net> wrote: 

TZ> Because HTTP is worse than HTTP/S as a software delivery channel in
TZ> almost every way.

IS> 	… Except for the cache-ability, where HTTP is a sure winner.

I don't think that matters too much in our case, but yes, that's one of
the ways HTTP is better.

Ted

Re: serving ELPA over HTTP/S

[Stefan Monnier]

> Clearly, because I think it's worth annoying the user, since plain HTTP
> should be the exception.

"Because I like it" is not a very compelling argument in this case.

SM> Why?  Why not just
SM> (if (we-have-gnutls) <thehttpsurl> <thehttpurl>)
> 2) because I don't think this should be set in the code.  It's a user choice.

We're talking about the expression that computes the default value of
a custom var.  So, of course, the user can override it in the usual way.

> 1) so ELPA archives can have multiple URLs. Assuming there's just one is
> not ideal in the long term.

That's a separate issue, unrelated to http/https.


        Stefan

Re: serving ELPA over HTTP/S

[Ted Zlatanov]

On Tue, 05 May 2015 07:50:14 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Clearly, because I think it's worth annoying the user, since plain HTTP
>> should be the exception.

SM> "Because I like it" is not a very compelling argument in this case.

That's not what I said.  I just didn't think I needed to explain why a
plaintext protocol is a bad choice here.  There are two issues:

1) all web traffic should be encrypted by default. Unencrypted traffic
is much easier to monitor, MITM, and DoS. You can find support for this
stance pretty easily, e.g.
http://blog.codinghorror.com/should-all-web-traffic-be-encrypted/ or in
the proposed HTTP 2.0 standard. You may disagree, but I think the burden
of proof today should be on those who want to *disable* encryption.

2) Specifically for Emacs, several attacks (e.g. those against package
signature verification that have been posted on the bugs list, like the
replay attack in
https://lists.gnu.org/archive/html/bug-gnu-emacs/2015-01/msg00002.html)
are dramatically easier against a plaintext transfer than against an
encrypted one. 

If the user doesn't have GnuPG installed (and we've agreed to treat that
as an acceptable situation, right?), encrypting the transfer is even
more crucial. It's too easy for the annoyed user to say "forget
verifying signatures, I want to get my work done" so we should try to at
least put some defenses around them.

This user laziness is not a theoretical problem. See for example
https://emacs.stackexchange.com/questions/233/how-to-proceed-on-package-el-signature-check-failure
where one of the suggestions is https://emacs.stackexchange.com/a/5502

    "I changed the package-check-signature value to nil and the problem
    was resolved."

Users are not going to look into why the key was missing and contact the
package author or inspect their HTTP traffic, they'll blaze ahead.

So I note again that my proposal is simply to warn the user, not to
block them from using plain HTTP. And, of course, they'll be able to
turn off the warning if they really don't want to encrypt their package
download channel. Is that really unreasonable and annoying?

SM> Why?  Why not just
SM> (if (we-have-gnutls) <thehttpsurl> <thehttpurl>)
>> 2) because I don't think this should be set in the code.  It's a user choice.

SM> We're talking about the expression that computes the default value of
SM> a custom var.  So, of course, the user can override it in the usual way.

The default you propose is *code*, determined at runtime (unless the
user customizes it). I think instead, *in this case*, we should always
try HTTP/S first and complain when it's not available, as I said above.
The user can then choose to silence all such warnings or change the default.

>> 1) so ELPA archives can have multiple URLs. Assuming there's just one is
>> not ideal in the long term.

SM> That's a separate issue, unrelated to http/https.

And yet it would also be addressed by my proposal, so I think it's worth
considering.

Please consider how many other package managers allow multiple URLs.  Is
there a real disadvantage to allowing a list for `package-archives'
entries, especially since it degrades gracefully to the current format?
Regardless of your opinion on HTTP/S, I think this is a useful feature.

Ted

Re: serving ELPA over HTTP/S

[Artur Malabarba]

> Please consider how many other package managers allow multiple URLs.  Is
> there a real disadvantage to allowing a list for `package-archives'
> entries, especially since it degrades gracefully to the current format?

The only disadvantage I can see would be an added layer of complexity
to the archive-refreshing code. Because there is now support for async
refreshing, it is not a simple matter of doing a loop over the urls.
I'm not saying it would be horribly complicated (in fact, I'm
perfectly willing to do it myself).

Re: serving ELPA over HTTP/S

[Stefan Monnier]

> http://blog.codinghorror.com/should-all-web-traffic-be-encrypted/ or in
> the proposed HTTP 2.0 standard. You may disagree, but I think the burden
> of proof today should be on those who want to *disable* encryption.

I largely agree, but at the same time, we've been running without even
any kind of signature verification until very recently, and even Debian
works without https, so clearly it's not that big of deal.

> If the user doesn't have GnuPG installed (and we've agreed to treat that
> as an acceptable situation, right?),

I could agree to emitting a warning if neither of gnutls nor gnupg
are available.
And I don't see a good reason to let the user turn the warning off
(after all, she can turn it off by installing gnupg).

>>> 1) so ELPA archives can have multiple URLs. Assuming there's just one is
>>> not ideal in the long term.
SM> That's a separate issue, unrelated to http/https.
> And yet it would also be addressed by my proposal, so I think it's worth
> considering.

I'm not opposed, but I think it's much more complex than just using
https by default when it's available.

Having several URL with a failover from one to the other, opens up the
issue of timeouts and other forms of failures, which can be pretty ugly,
so will require more care in the implementation to make it work well
enough (after defining what "well enough" should be in this context).


        Stefan