From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: "Christopher J. White" Newsgroups: gmane.emacs.bugs Subject: bug#9592: Problems with url-http, https, POSTS, reusing connections, and trailing CRLF after POST data Date: Sat, 24 Sep 2011 12:13:08 -0400 Message-ID: <20110924121308.00007b3c@unknown> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1316887979 23854 80.91.229.12 (24 Sep 2011 18:12:59 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sat, 24 Sep 2011 18:12:59 +0000 (UTC) To: 9592@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sat Sep 24 20:12:54 2011 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([140.186.70.17]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1R7Wib-0003Fr-3V for geb-bug-gnu-emacs@m.gmane.org; Sat, 24 Sep 2011 20:12:53 +0200 Original-Received: from localhost ([::1]:41240 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1R7Wia-00085z-J1 for geb-bug-gnu-emacs@m.gmane.org; Sat, 24 Sep 2011 14:12:52 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:38774) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1R7WhH-0007xk-0H for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 14:11:33 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1R7WhE-0001ZD-OU for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 14:11:30 -0400 Original-Received: from debbugs.gnu.org ([140.186.70.43]:48433) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1R7WhE-0001Z9-LW for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 14:11:28 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.69) (envelope-from ) id 1R7Whl-00069w-My for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 14:12:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: "Christopher J. White" Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 24 Sep 2011 18:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 9592 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.131688788823620 (code B ref -1); Sat, 24 Sep 2011 18:12:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 24 Sep 2011 18:11:28 +0000 Original-Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1R7WhC-00068s-Gh for submit@debbugs.gnu.org; Sat, 24 Sep 2011 14:11:28 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1R7UrU-0003T7-4q for submit@debbugs.gnu.org; Sat, 24 Sep 2011 12:13:58 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1R7Uqu-00009c-SN for submit@debbugs.gnu.org; Sat, 24 Sep 2011 12:13:23 -0400 Original-Received: from lists.gnu.org ([140.186.70.17]:34422) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1R7Uqu-00009Y-Qk for submit@debbugs.gnu.org; Sat, 24 Sep 2011 12:13:20 -0400 Original-Received: from eggs.gnu.org ([140.186.70.92]:35801) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1R7Uqs-0007Xx-M9 for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 12:13:20 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1R7Uqp-00008y-SX for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 12:13:18 -0400 Original-Received: from mail24c25.carrierzone.com ([64.29.147.34]:39993) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1R7Uqp-00008l-Gr for bug-gnu-emacs@gnu.org; Sat, 24 Sep 2011 12:13:15 -0400 X-Authenticated-User: cjwhite.grierwhite.com Original-Received: from unknown (pool-108-20-106-149.bstnma.east.verizon.net [108.20.106.149]) (authenticated bits=0) by mail24c25.carrierzone.com (8.13.6/8.13.1) with ESMTP id p8OGD9rg020393 for ; Sat, 24 Sep 2011 16:13:10 GMT X-Mailer: Claws Mail 3.7.8cvs63 (GTK+ 2.16.6; i586-pc-mingw32msvc) X-CSC: 0 X-CHA: v=1.1 cv=qPTRdQ0BYkhR73z0lvVjtSIhj4Pcdet57eUvyTqbk4M= c=1 sm=1 a=05zGyrpm-6EA:10 a=3pyGGIT-qAEA:10 a=kj9zAlcOel0A:10 a=xfvl1eDjdgK2iLzGOqA5FA==:17 a=mDV3o1hIAAAA:8 a=G0BRwk1oAAAA:8 a=yQBoeQYUg8xzW8cZ2uEA:9 a=Q8k4A9L1nMeoH7tlS9EA:7 a=CjuIK1q_8ugA:10 a=Uw-0TT0lQSwA:10 a=uyOpwfsfqbsA:10 a=Cs2AeJ6ydBb28gK4:21 a=zXtIoXVd653QsTPh:21 a=xfvl1eDjdgK2iLzGOqA5FA==:117 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Mailman-Approved-At: Sat, 24 Sep 2011 14:11:24 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Resent-Date: Sat, 24 Sep 2011 14:12:01 -0400 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 1) X-Received-From: 140.186.70.43 X-Mailman-Approved-At: Sat, 24 Sep 2011 14:12:49 -0400 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.bugs:51788 Archived-At: I've been having issues sending a POST requests to a server via https. The code works fine when using just http, but with https it does not work reliably. The first problem relates to bug #8931, terminating HTTP packets with extra CRLF. The patch for this bug undid a minor change that removed the CRLF after POST data. See: http://bzr.savannah.gnu.org/lh/emacs/trunk/revision/104908 The key change was: lisp/url/url-http.el: 340 ;; Any data 341 - url-http-data "\r\n")) 341 + url-http-data)) 342 "")) The added CRLF was put in because without it POSTs via HTTPS to at least some servers just hangs, apparently because gnutls-cli is not sending the url-http-data because of lack of CRLF. I found this problem myself, but also discovered that Mark Hershberger submitted a patch over a year ago to add the CRLF for the same reason. See: http://bzr.savannah.gnu.org/lh/emacs/trunk/revision/100681 I think a more appropriate fix is to only add the extra CRLF if POST data exists: 340 ;; Any data 341 - url-http-data)) 341 + url-http-data (if url-http-data "\r\n"))) 342 "")) I have been using this version above, with some success. However, I have now hit another stumbling block which is likely related. (BTW -- I tried to unarchive bug 8931 and amend the bug with the above details but I'm getting neither confirmation via email nor updates on debbugs.gnu.org. I sent the unarchive request to control debbugs.gnu.org) If I make a second HTTPS request after a POST with data, sometimes the second request attempts to reuse the old connection. I added some additional url-debug statements to isolate. (defun url-http-find-free-connection (host port) (url-http-debug "Finding free connction: %s:%d" host port) (url-http-debug "Currently open connections: %S" url-http-open-connections) ...) When it works, I see: http -> Finding free connction: api.toodledo.com:443 http -> Currently open connections: #s(hash-table size 17 test equal rehash-size 1.5 rehash-threshold 0.8 data (("api.toodledo.com" . 443) nil ("api.toodledo.com" . 80) nil)) http -> Contacting host: api.toodledo.com:443 http -> Marking connection as busy: api.toodledo.com:443 # http -> Request is: POST /2/tasks/get.php HTTP/1.1 MIME-Version: 1.0 Connection: keep-alive Extension: Security/Digest Security/SSL Host: api.toodledo.com Accept-charset: nil Accept: */* User-Agent: URL/Emacs (i386-mingw-nt6.1.7600; Windows-NT; 32bit) Content-Type: application/x-www-form-urlencoded Content-length: 187 retrieval -> Spinning in url-retrieve-synchronously: nil (#>) When it fails, I instead see: http -> Finding free connction: api.toodledo.com:443 http -> Currently open connections: #s(hash-table size 17 test equal rehash-size 1.5 rehash-threshold 0.8 data (("api.toodledo.com" . 443) (# #) ("api.toodledo.com" . 80) nil)) http -> Testing conn: #, process status run http -> Found existing connection: api.toodledo.com:443 # http -> Reusing existing connection: api.toodledo.com:443 http -> Marking connection as busy: api.toodledo.com:443 # http -> Request is: POST /2/tasks/get.php HTTP/1.1 MIME-Version: 1.0 Connection: keep-alive Extension: Security/Digest Security/SSL Host: api.toodledo.com Accept-charset: nil Accept: */* User-Agent: URL/Emacs (i386-mingw-nt6.1.7600; Windows-NT; 32bit) Content-Type: application/x-www-form-urlencoded Content-length: 57 http -> url-http-end-of-document-sentinel in buffer ( *http api.toodledo.com:443*<48>) http -> Marking connection as free: api.toodledo.com:443 # http -> Activating callback in buffer ( *http api.toodledo.com:443*<48>) retrieval -> Synchronous fetching done (#>) Along with the failure, I get the following backtrace: Debugger entered--Lisp error: (file-error "writing to process" "invalid argument" #) process-send-string(# "POST /2/tasks/deleted.php HTTP/1.1\nMIME-Version: 1.0\nConnection: keep-alive\nExtension: Security/Digest Security/SSL\nHost: api.toodledo.com\nAccept-charset: nil\nAccept: */*\nUser-Agent: URL/Emacs (i386-mingw-nt6.1.7600; Windows-NT; 32bit)\nContent-Type: application/x-www-form-urlencoded\nContent-length: 57\n\n\n") url-http([cl-struct-url "https" nil nil "api.toodledo.com" 443 "/2/tasks/get.php" nil nil t] ..... It's trying to reuse the old process as it is in the "run" state, but the process is not really usable. If I hack url-http to always create new connections (don't attempt reuse), things seem to work fine. The failure is fairly repeatable, but it does not happen precisely the same. I have a set of unit-tests for syncing org-mode tasks with toodledo.com, which makes on the order of 10s of POSTs to the server. It will fail every time, but not always at the same test. What's interesting is that if I put a 2 second delay before each POST, it seems to happen more regularly. I'm not sure where to take this from here. This is on a Window 7 box, running native emacs, but using gnu-tlscli from Cygwin. ...cj --- In GNU Emacs 23.3.1 (i386-mingw-nt6.1.7600) of 2011-03-10 on 3249CTO Windowing system distributor `Microsoft Corp.', version 6.1.7600 configured using `configure --with-gcc (4.5) --no-opt --cflags -Ic:/imagesupport/include' Important settings: value of $LC_ALL: nil value of $LC_COLLATE: nil value of $LC_CTYPE: nil value of $LC_MESSAGES: nil value of $LC_MONETARY: nil value of $LC_NUMERIC: nil value of $LC_TIME: nil value of $LANG: C.UTF-8 value of $XMODIFIERS: nil locale-coding-system: cp1252 default enable-multibyte-characters: t Major mode: Emacs-Lisp Minor modes in effect: show-paren-mode: t display-time-mode: t tooltip-mode: t mouse-wheel-mode: t global-font-lock-mode: t font-lock-mode: t auto-encryption-mode: t auto-compression-mode: t column-number-mode: t line-number-mode: t transient-mark-mode: t Recent input: C-] C-] C-] C-] C-] C-x b u r - h C-g ; ; C-x x x C-g C-x 1 C-x b C-x [ C-x b * M e s C-x ] C-x 1 C-x b o r g - . C-s C-g C-x [ C-s u s e - h t t p s C-s C-x x C-x [ C-x b * U * C-x ] C-a C-s c a l l - m e t h o d C-s C-s C-s C-s C-s C-a C-n C-n ( s l e e p p - o f f o r SPC 2 ) C-x x C-g C-] C-] C-] C-] x C-x [ x r e p SPC o SPC SPC r SPC SPC SPC Recent messages: All interaction with toodledo.com will be via HTTPS Contacting host: api.toodledo.com:443 Opening TLS connection to `api.toodledo.com'... Opening TLS connection with `gnutls-cli -p 443 api.toodledo.com'...done Opening TLS connection to `api.toodledo.com'...done Contacting host: api.toodledo.com:443 Entering debugger... Mark set Auto-saving...done Making completion list... [3 times] Load-path shadows: ~cwhite/lib/lisp/org-toodledo/w3mexcerpt hides ~cwhite/lib/lisp/w3mexcerpt ~cwhite/lib/lisp/org-toodledo/http-post-simple hides ~cwhite/lib/lisp/http-post-simple ~/lib/lisp/org-mode/lisp/org hides c:/Users/cwhite/emacs-23.3/lisp/org/org ~/lib/lisp/org-mode/lisp/org-xoxo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-xoxo ~/lib/lisp/org-mode/lisp/org-wl hides c:/Users/cwhite/emacs-23.3/lisp/org/org-wl ~/lib/lisp/org-mode/lisp/org-w3m hides c:/Users/cwhite/emacs-23.3/lisp/org/org-w3m ~/lib/lisp/org-mode/lisp/org-vm hides c:/Users/cwhite/emacs-23.3/lisp/org/org-vm ~/lib/lisp/org-mode/lisp/org-timer hides c:/Users/cwhite/emacs-23.3/lisp/org/org-timer ~/lib/lisp/org-mode/lisp/org-table hides c:/Users/cwhite/emacs-23.3/lisp/org/org-table ~/lib/lisp/org-mode/lisp/org-src hides c:/Users/cwhite/emacs-23.3/lisp/org/org-src ~/lib/lisp/org-mode/lisp/org-rmail hides c:/Users/cwhite/emacs-23.3/lisp/org/org-rmail ~/lib/lisp/org-mode/lisp/org-remember hides c:/Users/cwhite/emacs-23.3/lisp/org/org-remember ~/lib/lisp/org-mode/lisp/org-publish hides c:/Users/cwhite/emacs-23.3/lisp/org/org-publish ~/lib/lisp/org-mode/lisp/org-protocol hides c:/Users/cwhite/emacs-23.3/lisp/org/org-protocol ~/lib/lisp/org-mode/lisp/org-plot hides c:/Users/cwhite/emacs-23.3/lisp/org/org-plot ~/lib/lisp/org-mode/lisp/org-mouse hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mouse ~/lib/lisp/org-mode/lisp/org-mobile hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mobile ~/lib/lisp/org-mode/lisp/org-mhe hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mhe ~/lib/lisp/org-mode/lisp/org-mew hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mew ~/lib/lisp/org-mode/lisp/org-macs hides c:/Users/cwhite/emacs-23.3/lisp/org/org-macs ~/lib/lisp/org-mode/lisp/org-mac-message hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mac-message ~/lib/lisp/org-mode/lisp/org-list hides c:/Users/cwhite/emacs-23.3/lisp/org/org-list ~/lib/lisp/org-mode/lisp/org-latex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-latex ~/lib/lisp/org-mode/lisp/org-jsinfo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-jsinfo ~/lib/lisp/org-mode/lisp/org-irc hides c:/Users/cwhite/emacs-23.3/lisp/org/org-irc ~/lib/lisp/org-mode/lisp/org-install hides c:/Users/cwhite/emacs-23.3/lisp/org/org-install ~/lib/lisp/org-mode/lisp/org-inlinetask hides c:/Users/cwhite/emacs-23.3/lisp/org/org-inlinetask ~/lib/lisp/org-mode/lisp/org-info hides c:/Users/cwhite/emacs-23.3/lisp/org/org-info ~/lib/lisp/org-mode/lisp/org-indent hides c:/Users/cwhite/emacs-23.3/lisp/org/org-indent ~/lib/lisp/org-mode/lisp/org-id hides c:/Users/cwhite/emacs-23.3/lisp/org/org-id ~/lib/lisp/org-mode/lisp/org-icalendar hides c:/Users/cwhite/emacs-23.3/lisp/org/org-icalendar ~/lib/lisp/org-mode/lisp/org-html hides c:/Users/cwhite/emacs-23.3/lisp/org/org-html ~/lib/lisp/org-mode/lisp/org-habit hides c:/Users/cwhite/emacs-23.3/lisp/org/org-habit ~/lib/lisp/org-mode/lisp/org-gnus hides c:/Users/cwhite/emacs-23.3/lisp/org/org-gnus ~/lib/lisp/org-mode/lisp/org-freemind hides c:/Users/cwhite/emacs-23.3/lisp/org/org-freemind ~/lib/lisp/org-mode/lisp/org-footnote hides c:/Users/cwhite/emacs-23.3/lisp/org/org-footnote ~/lib/lisp/org-mode/lisp/org-feed hides c:/Users/cwhite/emacs-23.3/lisp/org/org-feed ~/lib/lisp/org-mode/lisp/org-faces hides c:/Users/cwhite/emacs-23.3/lisp/org/org-faces ~/lib/lisp/org-mode/lisp/org-exp hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp ~/lib/lisp/org-mode/lisp/org-exp-blocks hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp-blocks ~/lib/lisp/org-mode/lisp/org-docbook hides c:/Users/cwhite/emacs-23.3/lisp/org/org-docbook ~/lib/lisp/org-mode/lisp/org-datetree hides c:/Users/cwhite/emacs-23.3/lisp/org/org-datetree ~/lib/lisp/org-mode/lisp/org-crypt hides c:/Users/cwhite/emacs-23.3/lisp/org/org-crypt ~/lib/lisp/org-mode/lisp/org-compat hides c:/Users/cwhite/emacs-23.3/lisp/org/org-compat ~/lib/lisp/org-mode/lisp/org-colview hides c:/Users/cwhite/emacs-23.3/lisp/org/org-colview ~/lib/lisp/org-mode/lisp/org-clock hides c:/Users/cwhite/emacs-23.3/lisp/org/org-clock ~/lib/lisp/org-mode/lisp/org-bibtex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bibtex ~/lib/lisp/org-mode/lisp/org-bbdb hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bbdb ~/lib/lisp/org-mode/lisp/org-attach hides c:/Users/cwhite/emacs-23.3/lisp/org/org-attach ~/lib/lisp/org-mode/lisp/org-ascii hides c:/Users/cwhite/emacs-23.3/lisp/org/org-ascii ~/lib/lisp/org-mode/lisp/org-archive hides c:/Users/cwhite/emacs-23.3/lisp/org/org-archive ~/lib/lisp/org-mode/lisp/org-agenda hides c:/Users/cwhite/emacs-23.3/lisp/org/org-agenda Features: (shadow sort mail-extr message ecomplete rfc822 mml mml-sec password-cache mm-decode mm-bodies mm-encode mailabbrev nnheader gnus-util netrc gmm-utils mailheader canlock sha1 hex-util hashcash emacsbug apropos time-stamp grep find-func multi-isearch debug url-cache org-wl org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs org-html format-spec org-exp ob-exp org-exp-blocks org-agenda org-info org-gnus org-docview org-bibtex bibtex org-bbdb vc-git help-mode view org-toodledo w3mexcerpt remember org-remember org-datetree cygwin-mount ange-ftp uniquify filecache server paren time etags python-mode info-look css-mode scr rsync twiki edmacro kmacro info doku pib php-mode speedbar sb-image ezimage dframe assoc highline pop3 mail-utils cc-mode cc-fonts cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs cus-edit cus-start cus-load wid-edit exec http-post-simple cl cl-19 url-http tls url-auth mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums url-gw url url-proxy url-privacy url-expand url-methods url-history url-cookie url-util url-parse url-vars mm-util mail-prsvr mailcap json xml org byte-opt warnings advice help-fns advice-preload ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table org-footnote org-src ob-comint ob-keys ob ob-eval org-pcomplete pcomplete org-list org-faces org-compat org-entities org-macs time-date noutline outline easy-mmode regexp-opt cal-menu easymenu calendar cal-loaddefs compile comint ring bytecomp byte-compile tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32 disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button minibuffer faces cus-face files text-properties overlay md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote make-network-process multi-tty emacs)