`url-retrieve' for https behind proxy: 400 bad request

`url-retrieve' for https behind proxy: 400 bad request

[Andrew Savonichev]

Hi,

I use GNU/Emacs behind the http proxy with no user authorization, and
seems `url-retrieve' does not work correctly with the proxy-server -
error "400 Bad Request" for any https url.

Environment variables (http_proxy and https_proxy) are correctly set
and other tools except the emacs work fine: tried curl and wget.

I've compared url-retrieve and curl requests to the proxy-server
and there is a difference:
curl does CONNECT followed by GET, while url-retrieve use a direct
GET request.

curl (works fine):
  CONNECT www.google.com:443 HTTP/1.1
  Host: www.google.com:443
  User-Agent: curl/7.29.0
  Proxy-Connection: Keep-Alive

  GET / HTTP/1.1
  User-Agent: curl/7.29.0
  Host: www.google.com
  Accept: */*

url-retrieve (error 400):
  GET https://www.google.com/ HTTP/1.1
  MIME-Version: 1.0
  Extension: Security/Digest Security/SSL
  Host: www.google.com
  Accept-encoding: gzip
  Accept: */*
  User-Agent: URL/Emacs

I've tried it with the GNU/Emacs 25.1 and 24.3.1 releases (with -Q
command line option).

Is it a bug? Any ideas  would be appreciated.

    - Andrew

Re: `url-retrieve' for https behind proxy: 400 bad request

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Nov 08, 2016 at 09:18:07AM +0300, Andrew Savonichev wrote:
> Hi,
> 
> I use GNU/Emacs behind the http proxy with no user authorization, and
> seems `url-retrieve' does not work correctly with the proxy-server -
> error "400 Bad Request" for any https url.
> 
> Environment variables (http_proxy and https_proxy) are correctly set
> and other tools except the emacs work fine: tried curl and wget.
> 
> I've compared url-retrieve and curl requests to the proxy-server
> and there is a difference:
> curl does CONNECT followed by GET, while url-retrieve use a direct
> GET request.
> 
> curl (works fine):
>   CONNECT www.google.com:443 HTTP/1.1
>   Host: www.google.com:443
>   User-Agent: curl/7.29.0
>   Proxy-Connection: Keep-Alive
> 
>   GET / HTTP/1.1
>   User-Agent: curl/7.29.0
>   Host: www.google.com
>   Accept: */*
> 
> url-retrieve (error 400):
>   GET https://www.google.com/ HTTP/1.1
>   MIME-Version: 1.0
>   Extension: Security/Digest Security/SSL
>   Host: www.google.com
>   Accept-encoding: gzip
>   Accept: */*
>   User-Agent: URL/Emacs

Yeah. This is the usual dance for https over proxy (if the proxy allows
it). Basically, CONNECT tells the proxy to just pass the https stream
along, untouched.

> I've tried it with the GNU/Emacs 25.1 and 24.3.1 releases (with -Q
> command line option).

Seems url-retrieve hasn't implemented *that* wart yet. Consider
reporting a bug (Disclaimer: I haven't tested with latest Emacs).

> Is it a bug? Any ideas  would be appreciated.

I think so (but see disclaimer above). Or perhaps a missing feature.

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlghvdoACgkQBcgs9XrR2ka+wQCfelLpIhZtizQbUNWo/stlF1ld
Z+oAnivbrKHDq9OLp+H22DhwgqUKyVbv
=kWtF
-----END PGP SIGNATURE-----

Re: `url-retrieve' for https behind proxy: 400 bad request

[Yuri Khan]

On Tue, Nov 8, 2016 at 6:58 PM,  <tomas@tuxteam.de> wrote:

> Yeah. This is the usual dance for https over proxy (if the proxy allows
> it). Basically, CONNECT tells the proxy to just pass the https stream
> along, untouched.

And it’s the only way to preserve integrity of the connection. With
CONNECT, provided that the origin server presents a valid and matching
certificate and you check it, neither the proxy nor any other
man-in-the-middle between you and the proxy can spoof the origin
server, tamper with requests or responses, or sniff the traffic
between you and the origin. With “GET https://foo/bar”, all of the
above would be possible.

Re: `url-retrieve' for https behind proxy: 400 bad request

[David Engster]

'tomas' writes:
> Yeah. This is the usual dance for https over proxy (if the proxy allows
> it). Basically, CONNECT tells the proxy to just pass the https stream
> along, untouched.

This is implemented in Emacs master (but not available in Emacs 25,
unfortunately):

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=11788

-David

Re: `url-retrieve' for https behind proxy: 400 bad request

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Nov 08, 2016 at 05:26:22PM +0100, David Engster wrote:
> 'tomas' writes:
> > Yeah. This is the usual dance for https over proxy (if the proxy allows
> > it). Basically, CONNECT tells the proxy to just pass the https stream
> > along, untouched.
> 
> This is implemented in Emacs master (but not available in Emacs 25,
> unfortunately):
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=11788

Thanks for the info!

regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlgi44oACgkQBcgs9XrR2kauEwCfcoZiVj5EBCFFuPQAxFqRHzLz
fEMAnjRe4RUoZRDWKgIpm3G2u6we5L1h
=p8pA
-----END PGP SIGNATURE-----

Re: `url-retrieve' for https behind proxy: 400 bad request

[Andrew Savonichev]

> This is implemented in Emacs master (but not available in Emacs 25

Thank you! I've applied the patch to 25.1 and it works great!

    - Andrew