bug#16220: url-http.el: Not conforming to HTTP spec

[Stefan Monnier <monnier@iro.umontreal.ca>]

> The problem is that url-http.el sets a lot of headers by default that
> can not be overwritten in any other way then dynamically overshadowing
> some variables.

Indeed, this is ugly.  Improvements welcome.

> For example, all connections are keep-alive by
> default, which is confusing in itself already,

Not sure why it should be a problem.

> (let ((url-request-method "GET")
>       (url-request-extra-headers '(("Connection" . "close"))))
>   (url-retrieve-synchronously "http://www.google.com/"))
> And what is sent is this:
> GET / HTTP/1.1
> Connection: keep-alive
> ...
> Connection: close

> Which again isn't valid HTTP and the behaviour of the HTTP server in
> this case is undefined and implementation specific. The only way to
> workaround this is doing this:

> (let ((url-http-attempt-keepalives nil)
>       (url-request-method "GET")
>       (url-request-extra-headers '(("Connection" . "close"))))
>   (url-retrieve-synchronously "http://www.google.com/"))

Yuck!  We can probably fix this fairly easily by letting
url-request-extra-headers override (rather than just add to)
other headers.

> This is all the more irritating so many of the headers are set by
> default using the variables url-vars.el. Why those things are at all
> variables is a mystery to me.

Probably partly historical evolution (there was no place to add new
"parameters", so adding dynamic vars was an easy way to add more control
without breaking existing code).

> In the end it is much easier to do HTTP requests manually using
> make-network-process then it is with the url library,

I think that's misleading: the URL library is supposed to deal with
things like proxies and redirections, which "manual requests via
make-network-process" probably won't handle.

> Didn't anyone else run into problems with it?

Apparently not yet.  But I agree that the API might deserve a redesign
(IIRC another problem is in the way headers in the answer are returned
to the caller, which does not work consistently across different kinds
of URLs (ftp, http, file, imap, ...)).


        Stefan