From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: =?UTF-8?Q?Jaros=C5=82aw_?= =?UTF-8?Q?Rzesz=C3=B3tko?= Newsgroups: gmane.emacs.bugs Subject: bug#16220: url-http.el: Not conforming to HTTP spec Date: Sun, 22 Dec 2013 21:52:22 +0100 Message-ID: NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7bdc7c881f476f04ee25b0f4 X-Trace: ger.gmane.org 1387745584 31100 80.91.229.3 (22 Dec 2013 20:53:04 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 22 Dec 2013 20:53:04 +0000 (UTC) To: 16220@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sun Dec 22 21:53:09 2013 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Vuq1N-0002cY-Ae for geb-bug-gnu-emacs@m.gmane.org; Sun, 22 Dec 2013 21:53:09 +0100 Original-Received: from localhost ([::1]:59217 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vuq1M-0000Bd-Sc for geb-bug-gnu-emacs@m.gmane.org; Sun, 22 Dec 2013 15:53:08 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:52825) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vuq1I-0000BM-57 for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:53:05 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Vuq1G-0001pL-KW for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:53:04 -0500 Original-Received: from debbugs.gnu.org ([140.186.70.43]:49123) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vuq1G-0001pH-GT for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:53:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.80) (envelope-from ) id 1Vuq1G-0000VX-0b for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:53:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jaros=C5=82aw_?= =?UTF-8?Q?Rzesz=C3=B3tko?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 22 Dec 2013 20:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 16220 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.13877455561894 (code B ref -1); Sun, 22 Dec 2013 20:53:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 22 Dec 2013 20:52:36 +0000 Original-Received: from localhost ([127.0.0.1]:34909 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Vuq0p-0000UU-RX for submit@debbugs.gnu.org; Sun, 22 Dec 2013 15:52:36 -0500 Original-Received: from eggs.gnu.org ([208.118.235.92]:50994) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Vuq0m-0000UG-GS for submit@debbugs.gnu.org; Sun, 22 Dec 2013 15:52:33 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Vuq0k-0001mq-Ih for submit@debbugs.gnu.org; Sun, 22 Dec 2013 15:52:31 -0500 Original-Received: from lists.gnu.org ([2001:4830:134:3::11]:58543) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vuq0k-0001mm-Fv for submit@debbugs.gnu.org; Sun, 22 Dec 2013 15:52:30 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:52710) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vuq0i-0000A1-Vp for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:52:30 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Vuq0e-0001is-QO for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:52:28 -0500 Original-Received: from mail-pd0-x236.google.com ([2607:f8b0:400e:c02::236]:58670) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Vuq0e-0001g6-EJ for bug-gnu-emacs@gnu.org; Sun, 22 Dec 2013 15:52:24 -0500 Original-Received: by mail-pd0-f182.google.com with SMTP id v10so4437533pde.41 for ; Sun, 22 Dec 2013 12:52:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=3ov+rdqcAXD+gs/CbbH6F7tVL+MxLA93Cb0eIH7pAT8=; b=i45s1f4Evsq1N/WsiVYJXVj0o6o+6TekIgBHge0Y0Q49ThXlwNAWLu1DkhNrx3JooU EFtJjPEg11Rn4fvhvFksjvzDro1XGb0LEdnOC1aPnOCBuI7tmRCgzozm7DWNdqZ5c6Fp jknVGLbmMbOtUr0GPgLS1UiUKZY/7AD7yDgL787JJ3ZLH5AA6yq5aKuX91lO3kQn8z0Q kSMifovtBWIUAVIcQAjYiFKNLN+xBl7EPZuVpQx6JUEyDTq4wS7+al654MgWcJkIK+ws GSz62ztlwGEltd8LAAjjmXECpBkWwF19tEa+LPHk2P1lWtQxJrPe3RsCHC8m8rcB8tfS Oajw== X-Received: by 10.66.168.12 with SMTP id zs12mr22232651pab.122.1387745542881; Sun, 22 Dec 2013 12:52:22 -0800 (PST) Original-Received: by 10.66.77.230 with HTTP; Sun, 22 Dec 2013 12:52:22 -0800 (PST) X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 140.186.70.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:82406 Archived-At: --047d7bdc7c881f476f04ee25b0f4 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Hi, At the end of every HTTP request to be made with url-http.el and containing a body, an unnecessary "\r\n" is appended, and additionally those two characters are not used in the calculation of the Content-Length header. This normally would not matter, because a carefully build server will anyway only read "Content-Length" bytes from the body and ignore the final CRLF, but Emacs additionally defaults to using Connection: keep-alive, which results in the TCP traffic for what was meant to be a single request, being interpreted as two separate HTTP requests, the first one being roughly the intended one, and the other one consisting only of CRLF. In particular, I am using the HTTP server from net.http in Go language. That keepalive is enabled by default is strange, especially given how the variable that controls this is described: (defvar url-http-attempt-keepalives t "Whether to use a single TCP connection multiple times in HTTP. This is only useful when debugging the HTTP subsystem. Setting to nil will explicitly close the connection to the server after every request.") Those issues have been somewhat discussed here, but it seems the people discussing unfortunately don't understand HTTP: https://groups.google.com/forum/#!msg/gnu.emacs.bug/SF4P7gVI6IQ/SExtWzutKI4= J Please just compare this discussion to http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html If you don't go into any fancy things like chunked encoding etc., an HTTP request is a: - sequence of headers, each header followed with a newline - a newline terminating the sequence of headers - an optional request body The request body will be read by the server only if one of the headers is a Content-Length header, and the value of the header is exactly the number of bytes that is being sent, there is no CRLF terminating the body. So if there is no body, the request ends with two CRLFs, if there is a body, it just ends with [Content-Length] bytes of raw data. There is no possibility a proper HTTP server could be confused by a request being terminated with two CRLFs, if the request is otherwise correct. I think there must have been some confusion as to the reason of the original problem, that was then turned into this "fix". For reference, my code is roughly this, and as mentioned I am using net.http from the Go language on the other end: (defun server-command (command) (let* ((url-request-method "POST") (url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded"))) (url-request-data (concat (url-hexify-string "command") "=3D" (url-hexify-string (json-encode command)))) (result-buffer (url-retrieve-synchronously server-url)) (result (with-current-buffer result-buffer (goto-char (point-min)) (delete-region (point-min) (search-forward "\n\n")) (buffer-string)))) (kill-buffer result-buffer) result)) Normally I get from this function the contents of the first, "correct" response body from the server, but if I run it a few times in quick succession I additonally get the string "HTTP/1.1 400 Bad Request" at the end, which is actually the second HTTP response showing up at random in the buffer (altough it's consistently sent by the server every time, as I can see in a sniffer). Cheers, Jaros=B3aw Rzesz=F3tko --047d7bdc7c881f476f04ee25b0f4 Content-Type: text/html; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable
Hi,

At the end of every HTTP request to b= e made with url-http.el and containing a body, an unnecessary "\r\n&qu= ot; is appended, and additionally those two characters are not used in the = calculation of the Content-Length header. This normally would not matter, b= ecause a carefully build server will anyway only read "Content-Length&= quot; bytes from the body and ignore the final CRLF, but Emacs additionally= defaults to using Connection: keep-alive, which results in the TCP traffic= for what was meant to be a single request, being interpreted as two separa= te HTTP requests, the first one being roughly the intended one, and the oth= er one consisting only of CRLF. In particular, I am using the HTTP server f= rom net.http in Go language. That keepalive is enabled by default is strang= e, especially given how the variable that controls this is described:

(defvar url-http-attempt-keepalives t
=A0 "Whether to use = a single TCP connection multiple times in HTTP.
This is only useful when= debugging the HTTP subsystem.=A0 Setting to
nil will explicitly close t= he connection to the server after every
request.")

Those issues have been somewhat discussed here= , but it seems the people discussing unfortunately don't understand HTT= P:

https://groups.google.com/forum/#!msg/gnu.emacs.bug= /SF4P7gVI6IQ/SExtWzutKI4J

Please just compare this discussion to http://www.w3.org/Protocols/rf= c2616/rfc2616-sec4.html

If you don't go into any fancy thin= gs like chunked encoding etc., an HTTP request is a:

- sequence of headers, each header followed with a newline
- a newline terminating the sequence of headers
- an optio= nal request body

The request body will be read by the ser= ver only if one of the headers is a Content-Length header, and the value of= the header is exactly the number of bytes that is being sent, there is no = CRLF terminating the body.=A0 So if there is no body, the request ends with= two CRLFs, if there is a body, it just ends with [Content-Length] bytes of= raw data. There is no possibility a proper HTTP server could be confused b= y a request being terminated with two CRLFs, if the request is otherwise co= rrect. I think there must have been some confusion as to the reason of the = original problem, that was then turned into this "fix".

For reference, my code is roughly this, and as mentioned I a= m using net.http from the Go language on the other end:

(defun serve= r-command (command)
=A0 (let* ((url-request-method "POST")
=A0=A0=A0=A0=A0=A0=A0=A0 (url-request-extra-headers '(("Content-Ty= pe" . "application/x-www-form-urlencoded")))
=A0=A0=A0=A0= =A0=A0=A0=A0 (url-request-data (concat (url-hexify-string "command&quo= t;) "=3D" (url-hexify-string (json-encode command))))
=A0=A0=A0=A0=A0=A0=A0=A0 (result-buffer (url-retrieve-synchronously server-= url))
=A0=A0=A0=A0=A0=A0=A0=A0 (result
=A0=A0=A0=A0=A0=A0=A0=A0=A0 (w= ith-current-buffer result-buffer
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (goto= -char (point-min))
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (delete-region (poi= nt-min) (search-forward "\n\n"))
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 (buffer-string))))
=A0=A0=A0 (kill-buf= fer result-buffer)
=A0=A0=A0 result))

Normally I get from t= his function the contents of the first, "correct" response body f= rom the server, but if I run it a few times in quick succession I additonal= ly get the string "HTTP/1.1 400 Bad Request" at the end, which is= actually the second HTTP response showing up at random in the buffer (alto= ugh it's consistently sent by the server every time, as I can see in a = sniffer).

Cheers,
Jaros=B3aw Rzesz=F3tko
--047d7bdc7c881f476f04ee25b0f4--