bug#18860: 24.4; packages don't download consistently from https

[Nic Ferrier <nferrier@ferrier.me.uk>]

Emacs 24.4's package system does something odd when the archive is on
HTTPS.

It seems as if dependencies are downloaded through HTTPS but the main
package is attempted to be downloaded through HTTP.

Here's how to reproduce:

$ cat > test.el <<HERE
(let ((package-user-dir (make-temp-name "/tmp/emacs-package-bug")))
  (package-initialize)
  (add-to-list
   (quote package-archives)
   (quote ("marmalade" . "https://marmalade-repo.org/packages/")))
  (package-refresh-contents)
  (package-install (quote elpakit)))
HERE

$ emacs -batch -l test.el
Importing package-keyring.gpg...
Importing package-keyring.gpg...done
Contacting host: marmalade-repo.org:443
Contacting host: marmalade-repo.org:443
Contacting host: marmalade-repo.org:443
Making version-control local to s-autoloads.el while let-bound!
Generating autoloads for s.el...
Generating autoloads for s.el...done
Saving file /tmp/emacs-package-bug2503RFt/s-1.9.0/s-autoloads.el...
Wrote /tmp/emacs-package-bug2503RFt/s-1.9.0/s-autoloads.el
Checking /tmp/emacs-package-bug2503RFt/s-1.9.0...
Compiling /tmp/emacs-package-bug2503RFt/s-1.9.0/s-autoloads.el...
Compiling /tmp/emacs-package-bug2503RFt/s-1.9.0/s-pkg.el...
Wrote /tmp/emacs-package-bug2503RFt/s-1.9.0/s-pkg.elc
Compiling /tmp/emacs-package-bug2503RFt/s-1.9.0/s.el...
Wrote /tmp/emacs-package-bug2503RFt/s-1.9.0/s.elc
Done (Total of 2 files compiled, 1 skipped)
https://marmalade-repo.org/packages/noflet-0.0.14.el: Bad Request


It seems random which of these requests fail. But as soon as one is sent
over HTTP it fails (obviously).


marmalade-repo (which is currently the only repo doing https package
archives) could fix this problem, partially, on the server side. But we
can't protect the user that way. As soon as emacs makes an HTTP request
for something that should be signed the user is vulnerable to attack.

This is particularly egregious for a packaging system.




In GNU Emacs 24.4.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.23)
 of 2014-10-20 on nicferrier-XPS13-9333
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.1 LTS

Configured using:
 `configure --prefix=/home/nicferrier/emacs-24-4'

Important settings:
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix