From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: =?UTF-8?Q?Jaros=C5=82aw_?= =?UTF-8?Q?Rzesz=C3=B3tko?= Newsgroups: gmane.emacs.bugs Subject: bug#16220: url-http.el: Not conforming to HTTP spec Date: Tue, 24 Dec 2013 17:31:57 +0100 Message-ID: References: NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1387902733 20547 80.91.229.3 (24 Dec 2013 16:32:13 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 24 Dec 2013 16:32:13 +0000 (UTC) Cc: 16220@debbugs.gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Dec 24 17:32:18 2013 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1VvUu1-00019t-NP for geb-bug-gnu-emacs@m.gmane.org; Tue, 24 Dec 2013 17:32:17 +0100 Original-Received: from localhost ([::1]:39331 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VvUu1-00047U-AL for geb-bug-gnu-emacs@m.gmane.org; Tue, 24 Dec 2013 11:32:17 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:41275) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VvUts-00040T-Un for bug-gnu-emacs@gnu.org; Tue, 24 Dec 2013 11:32:13 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VvUtn-000291-9S for bug-gnu-emacs@gnu.org; Tue, 24 Dec 2013 11:32:08 -0500 Original-Received: from debbugs.gnu.org ([140.186.70.43]:53502) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VvUtn-00028u-5p for bug-gnu-emacs@gnu.org; Tue, 24 Dec 2013 11:32:03 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.80) (envelope-from ) id 1VvUtm-0005EL-CD for bug-gnu-emacs@gnu.org; Tue, 24 Dec 2013 11:32:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jaros=C5=82aw_?= =?UTF-8?Q?Rzesz=C3=B3tko?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 24 Dec 2013 16:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 16220 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 16220-submit@debbugs.gnu.org id=B16220.138790272020097 (code B ref 16220); Tue, 24 Dec 2013 16:32:02 +0000 Original-Received: (at 16220) by debbugs.gnu.org; 24 Dec 2013 16:32:00 +0000 Original-Received: from localhost ([127.0.0.1]:39288 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VvUtk-0005E3-38 for submit@debbugs.gnu.org; Tue, 24 Dec 2013 11:32:00 -0500 Original-Received: from mail-pd0-f178.google.com ([209.85.192.178]:49176) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VvUti-0005Du-3Z for 16220@debbugs.gnu.org; Tue, 24 Dec 2013 11:31:58 -0500 Original-Received: by mail-pd0-f178.google.com with SMTP id y10so6471663pdj.37 for <16220@debbugs.gnu.org>; Tue, 24 Dec 2013 08:31:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=/DHCHT+PpSLq079xr7eSEfW4t70A2eGfAnL2txcdJJ4=; b=a2gcjp2eHPxngS875tH0zU9Y2ZXJvha/govITk3XcQRX04alehj6WpNRPl2BWc1uqk bPQ44/scqPTzk/hOUQAYj6jIxED2p8JeqB27uQ3FmyHB2BGHBS9JVxoYJTh5Z5KeyVYD 6tU8ym6d56d2a7grh9Zf6QhrpYMtg2OK+VfF9bHP0o0B760+G+Tvg4PIKw1fiJ3Bo9jv nQpw3K+vGSOT1EFWHOSx0VpN6QlFQ457iYNsgCOwRK/GbeHYtTmJbz0NtUQpTDmW03eC dCDh4mmoiNrAOIe2y2TqajE+N0PzZVyXdanIQhOeVzUf2bKiRmPBmH0MGeaxLHcH3+Wh DqWg== X-Received: by 10.67.21.130 with SMTP id hk2mr33205100pad.76.1387902717148; Tue, 24 Dec 2013 08:31:57 -0800 (PST) Original-Received: by 10.66.77.230 with HTTP; Tue, 24 Dec 2013 08:31:57 -0800 (PST) In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 140.186.70.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:82529 Archived-At: Hi, There are more issues besides just the extra "\r\n", the url-http.el library is super confusing. I read the documentation for url-retrieve, which makes doing a HTTP request look really straightforward, and then I spent several hours finding out what the hell happens. For example, the documentation for url-retrieve says: "The variables `url-request-data', `url-request-method' and `url-request-extra-headers' can be dynamically bound around the request; dynamic binding of other variables doesn't necessarily take effect." The problem is that url-http.el sets a lot of headers by default that can not be overwritten in any other way then dynamically overshadowing some variables. For example, all connections are keep-alive by default, which is confusing in itself already, futhermore you can't just pass "Connection" . "close' in url-request-extra-headers, because url-http.el joins the default and "extra" headers instead of merging the "extra" ones and having them overwrite the default ones. So try do something like this: (let ((url-request-method "GET") (url-request-extra-headers '(("Connection" . "close")))) (url-retrieve-synchronously "http://www.google.com/")) And what is sent is this: GET / HTTP/1.1 Connection: keep-alive ... Connection: close Which again isn't valid HTTP and the behaviour of the HTTP server in this case is undefined and implementation specific. The only way to workaround this is doing this: (let ((url-http-attempt-keepalives nil) (url-request-method "GET") (url-request-extra-headers '(("Connection" . "close")))) (url-retrieve-synchronously "http://www.google.com/")) Which is only clear by reading url-http.el where again it is described in a very confusing way: (defvar url-http-attempt-keepalives t "Whether to use a single TCP connection multiple times in HTTP. This is only useful when debugging the HTTP subsystem. Setting to nil will explicitly close the connection to the server after every request.") This is all the more irritating so many of the headers are set by default using the variables url-vars.el. Why those things are at all variables is a mystery to me. Then there are messages that appear in the minibuffer and there is no way to disable them. In the end it is much easier to do HTTP requests manually using make-network-process then it is with the url library, it really isn't a good general purpose HTTP component, there are too many completely opaque things happening behind the scenes, like the hashmap of persistent tcp connections that is maintained behind the scenes. Didn't anyone else run into problems with it? Cheers, Jaros=B3aw Rzesz=F3tko 2013/12/24 Stefan Monnier : >> At the end of every HTTP request to be made with url-http.el and contain= ing >> a body, an unnecessary "\r\n" is appended, and additionally those two >> characters are not used in the calculation of the Content-Length header. > > Looks like an old workaround. Could someone get rid of this? > > > Stefan