bug#16220: url-http.el: Not conforming to HTTP spec

["Jarosław Rzeszótko" <sztywny@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

Hi,

At the end of every HTTP request to be made with url-http.el and containing
a body, an unnecessary "\r\n" is appended, and additionally those two
characters are not used in the calculation of the Content-Length header.
This normally would not matter, because a carefully build server will
anyway only read "Content-Length" bytes from the body and ignore the final
CRLF, but Emacs additionally defaults to using Connection: keep-alive,
which results in the TCP traffic for what was meant to be a single request,
being interpreted as two separate HTTP requests, the first one being
roughly the intended one, and the other one consisting only of CRLF. In
particular, I am using the HTTP server from net.http in Go language. That
keepalive is enabled by default is strange, especially given how the
variable that controls this is described:

(defvar url-http-attempt-keepalives t
  "Whether to use a single TCP connection multiple times in HTTP.
This is only useful when debugging the HTTP subsystem.  Setting to
nil will explicitly close the connection to the server after every
request.")

Those issues have been somewhat discussed here, but it seems the people
discussing unfortunately don't understand HTTP:

https://groups.google.com/forum/#!msg/gnu.emacs.bug/SF4P7gVI6IQ/SExtWzutKI4J

Please just compare this discussion to
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html

If you don't go into any fancy things like chunked encoding etc., an HTTP
request is a:

- sequence of headers, each header followed with a newline
- a newline terminating the sequence of headers
- an optional request body

The request body will be read by the server only if one of the headers is a
Content-Length header, and the value of the header is exactly the number of
bytes that is being sent, there is no CRLF terminating the body.  So if
there is no body, the request ends with two CRLFs, if there is a body, it
just ends with [Content-Length] bytes of raw data. There is no possibility
a proper HTTP server could be confused by a request being terminated with
two CRLFs, if the request is otherwise correct. I think there must have
been some confusion as to the reason of the original problem, that was then
turned into this "fix".

For reference, my code is roughly this, and as mentioned I am using
net.http from the Go language on the other end:

(defun server-command (command)
  (let* ((url-request-method "POST")
         (url-request-extra-headers '(("Content-Type" .
"application/x-www-form-urlencoded")))
         (url-request-data (concat (url-hexify-string "command") "="
(url-hexify-string (json-encode command))))
         (result-buffer (url-retrieve-synchronously server-url))
         (result
          (with-current-buffer result-buffer
            (goto-char (point-min))
            (delete-region (point-min) (search-forward "\n\n"))
            (buffer-string))))
    (kill-buffer result-buffer)
    result))

Normally I get from this function the contents of the first, "correct"
response body from the server, but if I run it a few times in quick
succession I additonally get the string "HTTP/1.1 400 Bad Request" at the
end, which is actually the second HTTP response showing up at random in the
buffer (altough it's consistently sent by the server every time, as I can
see in a sniffer).

Cheers,
Jarosław Rzeszótko

[-- Attachment #2: Type: text/html, Size: 3893 bytes --]