unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
@ 2012-02-09  4:53 Christopher J. White
  2012-02-09  9:15 ` Andreas Schwab
  2012-02-09 15:43 ` Andreas Schwab
  0 siblings, 2 replies; 8+ messages in thread
From: Christopher J. White @ 2012-02-09  4:53 UTC (permalink / raw)
  To: 10768

I’m using url-http to retrieve an XML response.  In noticed somewhat by
chance that certain page sizes were cutting off the last few characters
of the response, which in turn caused the XML to be poorly formed.

I've noticed the issue occurs when the payload response has to be sent
in two TCP packets and the last packet has 8 or fewer bytes.  I observe
this by looking at the HTTP stream in Wireshark.

I spent time tracing the response by adding some debug using
url-http-debug.  The first chunk of data comes from the first TCP packet
and includes the HTTP header and all the XML data minus the last 8 bytes.
url-http-wait-for-headers-change-function is called to process this
chunk of data.  After processing this chunk, the connection is closed
and the buffer of XML data returned minux the 8 bytes.

Here's the http trace, along with my added debug messages (inlcuding
dumping of the full data):

----------------------------------------
22:06.781000 http -> Contacting host: api.toodledo.com:80
22:06.827000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.828000 http -> Request is: 
POST /2/tasks/get.php HTTP/1.1 
MIME-Version: 1.0 
Connection: keep-alive 
Extension: Security/Digest Security/SSL 
Host: api.toodledo.com 
Accept-charset: nil 
Accept: */* 
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit) 
Content-Type: application/x-www-form-urlencoded 
Content-length: 212 
 
f=xml&key=9f2a92de083e8d63c2bd14771661139f&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0 

22:06.828000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer  *http api.toodledo.com:80*<80>>)
22:06.889000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
22:06.889000 http ->    data (point-max 1) (len 1380):
"HTTP/1.1 200 OK 
Date: Thu, 09 Feb 2012 04:22:06 GMT 
Server: Apache/2.2.3 (Red Hat) 
Content-Length: 1189 
Keep-Alive: timeout=2, max=100 
Connection: Keep-Alive 
Content-Type: application/xml 
 
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
22:06.889000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<80>)
22:06.890000 http -> Saw end of headers... ( *http api.toodledo.com:80*<80>) @ 197
22:06.890000 http -> url-http-clean-headers, end-of-headers now at 189
22:06.890000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<80>)
22:06.890000 http -> Got a content-length 1189, being smart about document end.
22:06.890000 http -> Calling initial content-length for extra data at end of headers
22:06.890000 http -> url-http-content-length-after-change-function 189 1381 1192
22:06.891000 http -> End of document found!
22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.891000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<80>)
22:06.891000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<80>)
22:06.891000 http -> Parsed HTTP headers: class=2 status=200
22:06.891000 http -> Finished parsing HTTP headers: t
22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.891000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<80>)
----------------------------------------

Notice that the XML in the data finishes with "...</note></task></".  It
should finish with:  "...</note></task></tasks>".  

A key debug message I added is "End of document found!", and this is
placed in the function url-http-content-length-after-change-function
right after the comment "Found the end of the document!  Wheee!":

(defun url-http-content-length-after-change-function (st nd length)
  <....>
  (if (> (- nd url-http-end-of-headers) url-http-content-length)
      (progn
	;; Found the end of the document!  Wheee!
        (url-http-debug "End of document found!")
	(url-display-percentage nil nil)
	(url-lazy-message "Reading... done.")
	(if (url-http-parse-headers)
	    (url-http-activate-callback)))))

If I slightly increase or decrease the amount of data returned
by the XML (by adding or deleting characters from title), I can get it
to work properly or not.  

For example, by adding 5 chars, I get:

----------------------------------------
39:20.144000 http -> Finding free connection: api.toodledo.com:80
39:20.144000 http -> Contacting host: api.toodledo.com:80
39:20.187000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.188000 http -> Request is: 
POST /2/tasks/get.php HTTP/1.1 
MIME-Version: 1.0 
Connection: keep-alive 
Extension: Security/Digest Security/SSL 
Host: api.toodledo.com 
Accept-charset: nil 
Accept: */* 
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit) 
Content-Type: application/x-www-form-urlencoded 
Content-length: 212 
 
f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0 

39:20.188000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer  *http api.toodledo.com:80*<86>>)
39:20.242000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
39:20.243000 http ->    data (point-max 1) (len 1380):
"HTTP/1.1 200 OK 
Date: Thu, 09 Feb 2012 04:39:20 GMT 
Server: Apache/2.2.3 (Red Hat) 
Content-Length: 1194 
Keep-Alive: timeout=2, max=100 
Connection: Keep-Alive 
Content-Type: application/xml 
 
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar12345</title><modified>1328762353</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></ta"
39:20.243000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<86>)
39:20.243000 http -> Saw end of headers... ( *http api.toodledo.com:80*<86>) @ 197
39:20.243000 http -> url-http-clean-headers, end-of-headers now at 189
39:20.243000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<86>)
39:20.243000 http -> Got a content-length 1194, being smart about document end.
39:20.243000 http -> Calling initial content-length for extra data at end of headers
39:20.243000 http -> url-http-content-length-after-change-function 189 1381 1192
39:20.244000 http -> Spinning waiting for headers...
39:20.244000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer  *http api.toodledo.com:80*<86>>)
39:20.247000 http -> Calling after change function `url-http-content-length-after-change-function' for `#<process api.toodledo.com<1>>'
39:20.247000 http ->    data (point-max 1373) (len 11):
"sk></tasks>"
39:20.247000 http -> url-http-content-length-after-change-function 1373 1384 11
39:20.247000 http -> End of document found!
39:20.247000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.247000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<86>)
39:20.247000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<86>)
39:20.247000 http -> Parsed HTTP headers: class=2 status=200
39:20.248000 http -> Finished parsing HTTP headers: t
39:20.248000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.248000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<86>)
----------------------------------------

In this case, url-http waits for the remaining data from the second TCP
packet, which comes in slightly later and is pieced together and forms a
good XML document.

If I reduce the size by an appropriate number of bytes so that the XML
data fits entirely in the first packet it also works. 

I think the problem relates to url-http-clean-headers.  I have 8-lines
of headers, and this function removes the trailing \r from each of those
lines.  I think this is not accounted for when using the content-length
to determine if all the data has been received.   After dropping these 8
trailing \r charaters, the buffer positions all change.  

I'm not quite able to figure out how the content-length is used to
detect the end of processing, so I thought I'd submit this bug and see
if any other folks can point me in the right direction.

...cj

In GNU Emacs 23.3.1 (i386-mingw-nt6.1.7601)
 of 2011-03-10 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 6.1.7601
configured using `configure --with-gcc (4.5) --no-opt --cflags -Ic:/imagesupport/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: C.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  show-paren-mode: t
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  global-font-lock-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
- - - - - - - - - - - - - - - - <return> <return> <switch-frame> 
<help-echo> <select-window> <backspace> <escape> C-x 
<select-window> <help-echo> C-/ C-/ C-/ C-/ SPC C-/ 
C-/ C-a <tab> <tab> C-x [ C-x ] C-x [ C-/ C-/ C-/ C-x 
[ C-SPC C-x ] C-w C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-a 
<tab> <S-tab> <tab> C-o s <help-echo> <switch-frame> 
C-SPC C-x ] <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <down> C-w <up> <down> <switch-frame> 
<switch-frame> <down> <down> C-x 2 <prior> <prior> 
<down> <down> <down-mouse-1> <mouse-movement> <mouse-1> 
<C-left> C-s C-w C-w C-w C-w C-w C-w C-w <return> C-x 
o C-s C-s C-a <help-echo> C-a C-l C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n <down> 
<down> <down> <down> C-x o C-p C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-l 
C-n C-n C-l C-n C-n C-n C-n C-n C-l C-n C-n C-n C-n 
C-l C-x o C-n C-n C-n <select-window> <select-window> 
<select-window> <help-echo> <prior> <prior> <next> 
<switch-frame> <down-mouse-1> <mouse-movement> <mouse-1> 
C-o s <select-window> <help-echo> <switch-frame> <down-mouse-1> 
<mouse-movement> <mouse-1> C-x ] <prior> <switch-frame> 
<down-mouse-1> <mouse-movement> <mouse-1> <down-mouse-1> 
<mouse-1> SPC <escape> C-x <select-window> <help-echo> 
<select-window> C-x o C-o s <switch-frame> <down-mouse-1> 
<mouse-movement> <mouse-1> C-x ] C-r t a s k s C-r 
C-r C-r C-s C-a <next> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <switch-frame> <switch-frame> 
<switch-frame> <switch-frame> <escape> x r e p o <tab> 
r t SPC e m SPC SPC <return>

Recent messages:
Reading [application/xml 
]... 1k of 1k (100%)
Contacting host: api.toodledo.com:80
Reading [application/xml 
]... 718 bytes of 717 bytes (100%)
Sync complete, 794 changes: recv 3 mod 791 del 
Mark set
Mark saved where search started
Auto-saving...done
Making completion list...

Load-path shadows:
~cwhite/lib/lisp/org-toodledo/w3mexcerpt hides ~cwhite/lib/lisp/w3mexcerpt
~cwhite/lib/lisp/org-toodledo/http-post-simple hides ~cwhite/lib/lisp/http-post-simple
~/lib/lisp/org-mode/lisp/org hides c:/Users/cwhite/emacs-23.3/lisp/org/org
~/lib/lisp/org-mode/lisp/org-xoxo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-xoxo
~/lib/lisp/org-mode/lisp/org-wl hides c:/Users/cwhite/emacs-23.3/lisp/org/org-wl
~/lib/lisp/org-mode/lisp/org-w3m hides c:/Users/cwhite/emacs-23.3/lisp/org/org-w3m
~/lib/lisp/org-mode/lisp/org-vm hides c:/Users/cwhite/emacs-23.3/lisp/org/org-vm
~/lib/lisp/org-mode/lisp/org-timer hides c:/Users/cwhite/emacs-23.3/lisp/org/org-timer
~/lib/lisp/org-mode/lisp/org-table hides c:/Users/cwhite/emacs-23.3/lisp/org/org-table
~/lib/lisp/org-mode/lisp/org-src hides c:/Users/cwhite/emacs-23.3/lisp/org/org-src
~/lib/lisp/org-mode/lisp/org-rmail hides c:/Users/cwhite/emacs-23.3/lisp/org/org-rmail
~/lib/lisp/org-mode/lisp/org-remember hides c:/Users/cwhite/emacs-23.3/lisp/org/org-remember
~/lib/lisp/org-mode/lisp/org-publish hides c:/Users/cwhite/emacs-23.3/lisp/org/org-publish
~/lib/lisp/org-mode/lisp/org-protocol hides c:/Users/cwhite/emacs-23.3/lisp/org/org-protocol
~/lib/lisp/org-mode/lisp/org-plot hides c:/Users/cwhite/emacs-23.3/lisp/org/org-plot
~/lib/lisp/org-mode/lisp/org-mouse hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mouse
~/lib/lisp/org-mode/lisp/org-mobile hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mobile
~/lib/lisp/org-mode/lisp/org-mhe hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mhe
~/lib/lisp/org-mode/lisp/org-mew hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mew
~/lib/lisp/org-mode/lisp/org-macs hides c:/Users/cwhite/emacs-23.3/lisp/org/org-macs
~/lib/lisp/org-mode/lisp/org-mac-message hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mac-message
~/lib/lisp/org-mode/lisp/org-list hides c:/Users/cwhite/emacs-23.3/lisp/org/org-list
~/lib/lisp/org-mode/lisp/org-latex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-latex
~/lib/lisp/org-mode/lisp/org-jsinfo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-jsinfo
~/lib/lisp/org-mode/lisp/org-irc hides c:/Users/cwhite/emacs-23.3/lisp/org/org-irc
~/lib/lisp/org-mode/lisp/org-install hides c:/Users/cwhite/emacs-23.3/lisp/org/org-install
~/lib/lisp/org-mode/lisp/org-inlinetask hides c:/Users/cwhite/emacs-23.3/lisp/org/org-inlinetask
~/lib/lisp/org-mode/lisp/org-info hides c:/Users/cwhite/emacs-23.3/lisp/org/org-info
~/lib/lisp/org-mode/lisp/org-indent hides c:/Users/cwhite/emacs-23.3/lisp/org/org-indent
~/lib/lisp/org-mode/lisp/org-id hides c:/Users/cwhite/emacs-23.3/lisp/org/org-id
~/lib/lisp/org-mode/lisp/org-icalendar hides c:/Users/cwhite/emacs-23.3/lisp/org/org-icalendar
~/lib/lisp/org-mode/lisp/org-html hides c:/Users/cwhite/emacs-23.3/lisp/org/org-html
~/lib/lisp/org-mode/lisp/org-habit hides c:/Users/cwhite/emacs-23.3/lisp/org/org-habit
~/lib/lisp/org-mode/lisp/org-gnus hides c:/Users/cwhite/emacs-23.3/lisp/org/org-gnus
~/lib/lisp/org-mode/lisp/org-freemind hides c:/Users/cwhite/emacs-23.3/lisp/org/org-freemind
~/lib/lisp/org-mode/lisp/org-footnote hides c:/Users/cwhite/emacs-23.3/lisp/org/org-footnote
~/lib/lisp/org-mode/lisp/org-feed hides c:/Users/cwhite/emacs-23.3/lisp/org/org-feed
~/lib/lisp/org-mode/lisp/org-faces hides c:/Users/cwhite/emacs-23.3/lisp/org/org-faces
~/lib/lisp/org-mode/lisp/org-exp hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp
~/lib/lisp/org-mode/lisp/org-exp-blocks hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp-blocks
~/lib/lisp/org-mode/lisp/org-docbook hides c:/Users/cwhite/emacs-23.3/lisp/org/org-docbook
~/lib/lisp/org-mode/lisp/org-datetree hides c:/Users/cwhite/emacs-23.3/lisp/org/org-datetree
~/lib/lisp/org-mode/lisp/org-crypt hides c:/Users/cwhite/emacs-23.3/lisp/org/org-crypt
~/lib/lisp/org-mode/lisp/org-compat hides c:/Users/cwhite/emacs-23.3/lisp/org/org-compat
~/lib/lisp/org-mode/lisp/org-colview hides c:/Users/cwhite/emacs-23.3/lisp/org/org-colview
~/lib/lisp/org-mode/lisp/org-clock hides c:/Users/cwhite/emacs-23.3/lisp/org/org-clock
~/lib/lisp/org-mode/lisp/org-bibtex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bibtex
~/lib/lisp/org-mode/lisp/org-bbdb hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bbdb
~/lib/lisp/org-mode/lisp/org-attach hides c:/Users/cwhite/emacs-23.3/lisp/org/org-attach
~/lib/lisp/org-mode/lisp/org-ascii hides c:/Users/cwhite/emacs-23.3/lisp/org/org-ascii
~/lib/lisp/org-mode/lisp/org-archive hides c:/Users/cwhite/emacs-23.3/lisp/org/org-archive
~/lib/lisp/org-mode/lisp/org-agenda hides c:/Users/cwhite/emacs-23.3/lisp/org/org-agenda

Features:
(shadow sort mail-extr message ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailabbrev nnheader
gnus-util netrc gmm-utils mailheader canlock sha1 hex-util hashcash
emacsbug sgml-mode cl-specs edebug find-func vc-git nxml-uchnm rng-xsd
xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok browse-url
cperl-mode url-cache org-colview diary-lib diary-loaddefs cal-iso
org-table bookmark pp image-file mule-util grep sh-script executable
tabify org-ascii help-mode view org-wl org-w3m org-vm org-rmail org-mhe
org-mew org-irc org-jsinfo org-infojs org-html format-spec org-exp
ob-exp org-exp-blocks org-agenda org-info org-gnus org-docview
org-bibtex bibtex org-bbdb multi-isearch newcomment org-toodledo
http-post-simple url-http tls url-auth mail-parse rfc2231 rfc2047
rfc2045 qp ietf-drums url-gw url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-util url-parse url-vars mm-util
mail-prsvr mailcap json xml w3mexcerpt remember org-remember
org-datetree org ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table
org-footnote org-src ob-comint ob-keys ob ob-eval org-pcomplete
pcomplete org-list org-faces org-compat org-entities org-macs time-date
noutline outline easy-mmode cal-menu calendar cal-loaddefs cygwin-mount
ange-ftp uniquify filecache server byte-opt warnings advice help-fns
advice-preload paren time etags python-mode info-look css-mode scr rsync
twiki edmacro kmacro info doku pib php-mode speedbar sb-image ezimage
dframe assoc highline pop3 mail-utils cl cl-19 cc-mode cc-fonts cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs regexp-opt cus-edit
easymenu cus-start cus-load wid-edit exec compile comint ring bytecomp
byte-compile tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32
disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty emacs)





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09  4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White
@ 2012-02-09  9:15 ` Andreas Schwab
  2012-02-09 12:49   ` Christopher J. White
  2012-02-09 15:43 ` Andreas Schwab
  1 sibling, 1 reply; 8+ messages in thread
From: Andreas Schwab @ 2012-02-09  9:15 UTC (permalink / raw)
  To: Christopher J. White; +Cc: 10768

"Christopher J. White" <chris@grierwhite.com> writes:

> "HTTP/1.1 200 OK 
> Date: Thu, 09 Feb 2012 04:22:06 GMT 
> Server: Apache/2.2.3 (Red Hat) 
> Content-Length: 1189 
                  ^^^^
> Keep-Alive: timeout=2, max=100 
> Connection: Keep-Alive 
> Content-Type: application/xml 
>  
> <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"

1189 bytes of data follow.

That looks like a bug in the server.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09  9:15 ` Andreas Schwab
@ 2012-02-09 12:49   ` Christopher J. White
  2012-02-09 13:07     ` Andreas Schwab
  0 siblings, 1 reply; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 12:49 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: 10768

On Thu, 09 Feb 2012 10:15:47 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:

> "Christopher J. White" <chris@grierwhite.com> writes:
> > Content-Length: 1189 
>                   ^^^^
> > <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3'
> > total='3'><task><id>9105869</id><title>Test
> > 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test
> > 2
> > -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
> 
> 1189 bytes of data follow.
> 
> That looks like a bug in the server.

I considered that and looked hard for it to be the server.  In the data
output the double quote chars add an additional slash "\" that is not
present in the actual data.  The content-length and the actual number
of raw bytes matches precisely when decoded from wireshark. 

I reran the test and collected the 3 sets of data, the recombined HTTM
response as built by wireshark (TCP Follow Stream), the ascii dump
of the wireshark packets (2 involved), and the URL-Debug.  The key point
is that the data in the second packet is never processed by url-http, only 
the bytes in the first packet.

HTTP response as seen in Wireshark:

----------------------------------------
f=xml&key=444ed11d926131624b2ce2ed81f58ad4&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 12:34:49 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1189
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml

<?xml version="1.0" encoding="UTF-8"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328790882</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><not
 e></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</prio!
 rity><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></tasks>
----------------------------------------

There is no newline at the end, the last character is the ">" after tasks.  
The xml data is all a single line, there are no CR/LF characters any where 
in it, so if any show up in this message they should be removed if you're
trying to count it out.

Here's the wireshark decode for the 2 packets involved:

----------------------------------------
No.     Time        Source                Destination           Protocol Length Info
    312 49.870066   72.4.112.215          11.1.1.89             TCP      1434   [TCP segment of a reassembled PDU]

Frame 312: 1434 bytes on wire (11472 bits), 1434 bytes captured (11472 bits)
    Arrival Time: Feb  9, 2012 07:34:49.261829000 Eastern Standard Time
    Epoch Time: 1328790889.261829000 seconds
    [Time delta from previous captured frame: 0.008369000 seconds]
    [Time delta from previous displayed frame: 0.008369000 seconds]
    [Time since reference or first frame: 49.870066000 seconds]
    Frame Number: 312
    Frame Length: 1434 bytes (11472 bits)
    Capture Length: 1434 bytes (11472 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:tcp]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80]
Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
    Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
        Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
        Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 1420
    Identification: 0xe548 (58696)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 55
    Protocol: TCP (6)
    Header checksum: 0x93ee [correct]
        [Good: True]
        [Bad: False]
    Source: 72.4.112.215 (72.4.112.215)
    Destination: 11.1.1.89 (11.1.1.89)
Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1, Ack: 527, Len: 1380
    Source port: http (80)
    Destination port: 60823 (60823)
    [Stream index: 10]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 1381    (relative sequence number)]
    Acknowledgement number: 527    (relative ack number)
    Header length: 20 bytes
    Flags: 0x10 (ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 54
    [Calculated window size: 6912]
    [Window size scaling factor: 128]
    Checksum: 0x836f [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
        [Bytes in flight: 1380]
    TCP segment data (1380 bytes)

0000  00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00   .!k^.....D....E.
0010  05 8c e5 48 40 00 37 06 93 ee 48 04 70 d7 0b 01   ...H@.7...H.p...
0020  01 59 00 50 ed 97 30 cd 5f f0 eb ef 69 24 50 10   .Y.P..0._...i$P.
0030  00 36 83 6f 00 00 48 54 54 50 2f 31 2e 31 20 32   .6.o..HTTP/1.1 2
0040  30 30 20 4f 4b 0d 0a 44 61 74 65 3a 20 54 68 75   00 OK..Date: Thu
0050  2c 20 30 39 20 46 65 62 20 32 30 31 32 20 31 32   , 09 Feb 2012 12
0060  3a 33 34 3a 34 39 20 47 4d 54 0d 0a 53 65 72 76   :34:49 GMT..Serv
0070  65 72 3a 20 41 70 61 63 68 65 2f 32 2e 32 2e 33   er: Apache/2.2.3
0080  20 28 52 65 64 20 48 61 74 29 0d 0a 43 6f 6e 74    (Red Hat)..Cont
0090  65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 31 38 39   ent-Length: 1189
00a0  0d 0a 4b 65 65 70 2d 41 6c 69 76 65 3a 20 74 69   ..Keep-Alive: ti
00b0  6d 65 6f 75 74 3d 32 2c 20 6d 61 78 3d 31 30 30   meout=2, max=100
00c0  0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 65   ..Connection: Ke
00d0  65 70 2d 41 6c 69 76 65 0d 0a 43 6f 6e 74 65 6e   ep-Alive..Conten
00e0  74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74   t-Type: applicat
00f0  69 6f 6e 2f 78 6d 6c 0d 0a 0d 0a 3c 3f 78 6d 6c   ion/xml....<?xml
0100  20 76 65 72 73 69 6f 6e 3d 22 31 2e 30 22 20 65    version="1.0" e
0110  6e 63 6f 64 69 6e 67 3d 22 55 54 46 2d 38 22 3f   ncoding="UTF-8"?
0120  3e 3c 74 61 73 6b 73 20 6e 75 6d 3d 27 33 27 20   ><tasks num='3' 
0130  74 6f 74 61 6c 3d 27 33 27 3e 3c 74 61 73 6b 3e   total='3'><task>
0140  3c 69 64 3e 39 31 30 35 38 36 39 3c 2f 69 64 3e   <id>9105869</id>
0150  3c 74 69 74 6c 65 3e 54 65 73 74 20 31 3c 2f 74   <title>Test 1</t
0160  69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e 31   itle><modified>1
0170  33 32 38 34 38 32 39 33 38 3c 2f 6d 6f 64 69 66   328482938</modif
0180  69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e 31   ied><completed>1
0190  33 32 37 39 32 34 38 30 30 3c 2f 63 6f 6d 70 6c   327924800</compl
01a0  65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f   eted><folder>0</
01b0  66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e   folder><context>
01c0  30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c   0</context><goal
01d0  3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69   >0</goal><priori
01e0  74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e 3c   ty>0</priority><
01f0  72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e   repeat></repeat>
0200  3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72   <repeatfrom>0</r
0210  65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61   epeatfrom><dueda
0220  74 65 3e 31 33 32 38 30 39 37 36 30 30 3c 2f 64   te>1328097600</d
0230  75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74   uedate><startdat
0240  65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c   e>0</startdate><
0250  64 75 65 74 69 6d 65 3e 31 33 32 38 31 33 37 32   duetime>13281372
0260  30 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61   00</duetime><sta
0270  72 74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74   rttime>0</startt
0280  69 6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c   ime><length>0</l
0290  65 6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c   ength><status>2<
02a0  2f 73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f   /status><note></
02b0  6e 6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73   note></task><tas
02c0  6b 3e 3c 69 64 3e 39 31 30 38 33 30 39 3c 2f 69   k><id>9108309</i
02d0  64 3e 3c 74 69 74 6c 65 3e 54 65 73 74 20 32 20   d><title>Test 2 
02e0  2d 66 6f 6f 62 61 72 3c 2f 74 69 74 6c 65 3e 3c   -foobar</title><
02f0  6d 6f 64 69 66 69 65 64 3e 31 33 32 38 37 39 30   modified>1328790
0300  38 38 32 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63   882</modified><c
0310  6f 6d 70 6c 65 74 65 64 3e 30 3c 2f 63 6f 6d 70   ompleted>0</comp
0320  6c 65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c   leted><folder>0<
0330  2f 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74   /folder><context
0340  3e 30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61   >0</context><goa
0350  6c 3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72   l>0</goal><prior
0360  69 74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e   ity>0</priority>
0370  3c 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74   <repeat></repeat
0380  3e 3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f   ><repeatfrom>0</
0390  72 65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64   repeatfrom><dued
03a0  61 74 65 3e 30 3c 2f 64 75 65 64 61 74 65 3e 3c   ate>0</duedate><
03b0  73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73 74 61   startdate>0</sta
03c0  72 74 64 61 74 65 3e 3c 64 75 65 74 69 6d 65 3e   rtdate><duetime>
03d0  30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 72   0</duetime><star
03e0  74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74 69   ttime>0</startti
03f0  6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65   me><length>0</le
0400  6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c 2f   ngth><status>2</
0410  73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e   status><note></n
0420  6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73 6b   ote></task><task
0430  3e 3c 69 64 3e 39 31 34 39 33 33 33 3c 2f 69 64   ><id>9149333</id
0440  3e 3c 74 69 74 6c 65 3e 6f 72 67 2d 74 6f 6f 64   ><title>org-tood
0450  6c 65 64 6f 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64   ledo</title><mod
0460  69 66 69 65 64 3e 31 33 32 38 30 31 31 39 34 39   ified>1328011949
0470  3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70   </modified><comp
0480  6c 65 74 65 64 3e 30 3c 2f 63 6f 6d 70 6c 65 74   leted>0</complet
0490  65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f   ed><folder>0</fo
04a0  6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c   lder><context>0<
04b0  2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30   /context><goal>0
04c0  3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79   </goal><priority
04d0  3e 32 3c 2f 70 72 69 6f 72 69 74 79 3e 3c 72 65   >2</priority><re
04e0  70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e 3c 72   peat></repeat><r
04f0  65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72 65 70   epeatfrom>0</rep
0500  65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61 74 65   eatfrom><duedate
0510  3e 30 3c 2f 64 75 65 64 61 74 65 3e 3c 73 74 61   >0</duedate><sta
0520  72 74 64 61 74 65 3e 30 3c 2f 73 74 61 72 74 64   rtdate>0</startd
0530  61 74 65 3e 3c 64 75 65 74 69 6d 65 3e 30 3c 2f   ate><duetime>0</
0540  64 75 65 74 69 6d 65 3e 3c 73 74 61 72 74 74 69   duetime><startti
0550  6d 65 3e 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e   me>0</starttime>
0560  3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74   <length>0</lengt
0570  68 3e 3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61   h><status>2</sta
0580  74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65   tus><note></note
0590  3e 3c 2f 74 61 73 6b 3e 3c 2f                     ></task></

No.     Time        Source                Destination           Protocol Length Info
    313 49.874992   72.4.112.215          11.1.1.89             HTTP/XML 60     HTTP/1.1 200 OK 

Frame 313: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
    Arrival Time: Feb  9, 2012 07:34:49.266755000 Eastern Standard Time
    Epoch Time: 1328790889.266755000 seconds
    [Time delta from previous captured frame: 0.004926000 seconds]
    [Time delta from previous displayed frame: 0.004926000 seconds]
    [Time since reference or first frame: 49.874992000 seconds]
    Frame Number: 313
    Frame Length: 60 bytes (480 bits)
    Capture Length: 60 bytes (480 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:tcp:http:xml]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80]
Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
    Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
        Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
        Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 46
    Identification: 0xe549 (58697)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 55
    Protocol: TCP (6)
    Header checksum: 0x994b [correct]
        [Good: True]
        [Bad: False]
    Source: 72.4.112.215 (72.4.112.215)
    Destination: 11.1.1.89 (11.1.1.89)
Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1381, Ack: 527, Len: 6
    Source port: http (80)
    Destination port: 60823 (60823)
    [Stream index: 10]
    Sequence number: 1381    (relative sequence number)
    [Next sequence number: 1387    (relative sequence number)]
    Acknowledgement number: 527    (relative ack number)
    Header length: 20 bytes
    Flags: 0x18 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 54
    [Calculated window size: 6912]
    [Window size scaling factor: 128]
    Checksum: 0xb632 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    [SEQ/ACK analysis]
        [Bytes in flight: 1386]
    TCP segment data (6 bytes)
[2 Reassembled TCP Segments (1386 bytes): #312(1380), #313(6)]
    [Frame: 312, payload: 0-1379 (1380 bytes)]
    [Frame: 313, payload: 1380-1385 (6 bytes)]
    [Segment count: 2]
    [Reassembled TCP length: 1386]
Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
            [Message: HTTP/1.1 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Status Code: 200
        Response Phrase: OK
    Date: Thu, 09 Feb 2012 12:34:49 GMT\r\n
    Server: Apache/2.2.3 (Red Hat)\r\n
    Content-Length: 1189\r\n
        [Content length: 1189]
    Keep-Alive: timeout=2, max=100\r\n
    Connection: Keep-Alive\r\n
    Content-Type: application/xml\r\n
    \r\n
eXtensible Markup Language
    <?xml
        version="1.0"
        encoding="UTF-8"
        ?>
    <tasks
        num='3'
        total='3'>
        <task>
            <id>
                9105869
                </id>
            <title>
                Test 1
                </title>
            <modified>
                1328482938
                </modified>
            <completed>
                1327924800
                </completed>
            <folder>
                0
                </folder>
            <context>
                0
                </context>
            <goal>
                0
                </goal>
            <priority>
                0
                </priority>
            <repeat>
                </repeat>
            <repeatfrom>
                0
                </repeatfrom>
            <duedate>
                1328097600
                </duedate>
            <startdate>
                0
                </startdate>
            <duetime>
                1328137200
                </duetime>
            <starttime>
                0
                </starttime>
            <length>
                0
                </length>
            <status>
                2
                </status>
            <note>
                </note>
            </task>
        <task>
            <id>
                9108309
                </id>
            <title>
                Test 2 -foobar
                </title>
            <modified>
                1328790882
                </modified>
            <completed>
                0
                </completed>
            <folder>
                0
                </folder>
            <context>
                0
                </context>
            <goal>
                0
                </goal>
            <priority>
                0
                </priority>
            <repeat>
                </repeat>
            <repeatfrom>
                0
                </repeatfrom>
            <duedate>
                0
                </duedate>
            <startdate>
                0
                </startdate>
            <duetime>
                0
                </duetime>
            <starttime>
                0
                </starttime>
            <length>
                0
                </length>
            <status>
                2
                </status>
            <note>
                </note>
            </task>
        <task>
            <id>
                9149333
                </id>
            <title>
                org-toodledo
                </title>
            <modified>
                1328011949
                </modified>
            <completed>
                0
                </completed>
            <folder>
                0
                </folder>
            <context>
                0
                </context>
            <goal>
                0
                </goal>
            <priority>
                2
                </priority>
            <repeat>
                </repeat>
            <repeatfrom>
                0
                </repeatfrom>
            <duedate>
                0
                </duedate>
            <startdate>
                0
                </startdate>
            <duetime>
                0
                </duetime>
            <starttime>
                0
                </starttime>
            <length>
                0
                </length>
            <status>
                2
                </status>
            <note>
                </note>
            </task>
        </tasks>

Frame (60 bytes):

0000  00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00   .!k^.....D....E.
0010  00 2e e5 49 40 00 37 06 99 4b 48 04 70 d7 0b 01   ...I@.7..KH.p...
0020  01 59 00 50 ed 97 30 cd 65 54 eb ef 69 24 50 18   .Y.P..0.eT..i$P.
0030  00 36 b6 32 00 00 74 61 73 6b 73 3e               .6.2..tasks>

Reassembled TCP (1386 bytes):

0000  48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d   HTTP/1.1 200 OK.
0010  0a 44 61 74 65 3a 20 54 68 75 2c 20 30 39 20 46   .Date: Thu, 09 F
0020  65 62 20 32 30 31 32 20 31 32 3a 33 34 3a 34 39   eb 2012 12:34:49
0030  20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70    GMT..Server: Ap
0040  61 63 68 65 2f 32 2e 32 2e 33 20 28 52 65 64 20   ache/2.2.3 (Red 
0050  48 61 74 29 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65   Hat)..Content-Le
0060  6e 67 74 68 3a 20 31 31 38 39 0d 0a 4b 65 65 70   ngth: 1189..Keep
0070  2d 41 6c 69 76 65 3a 20 74 69 6d 65 6f 75 74 3d   -Alive: timeout=
0080  32 2c 20 6d 61 78 3d 31 30 30 0d 0a 43 6f 6e 6e   2, max=100..Conn
0090  65 63 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69   ection: Keep-Ali
00a0  76 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65   ve..Content-Type
00b0  3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d   : application/xm
00c0  6c 0d 0a 0d 0a 3c 3f 78 6d 6c 20 76 65 72 73 69   l....<?xml versi
00d0  6f 6e 3d 22 31 2e 30 22 20 65 6e 63 6f 64 69 6e   on="1.0" encodin
00e0  67 3d 22 55 54 46 2d 38 22 3f 3e 3c 74 61 73 6b   g="UTF-8"?><task
00f0  73 20 6e 75 6d 3d 27 33 27 20 74 6f 74 61 6c 3d   s num='3' total=
0100  27 33 27 3e 3c 74 61 73 6b 3e 3c 69 64 3e 39 31   '3'><task><id>91
0110  30 35 38 36 39 3c 2f 69 64 3e 3c 74 69 74 6c 65   05869</id><title
0120  3e 54 65 73 74 20 31 3c 2f 74 69 74 6c 65 3e 3c   >Test 1</title><
0130  6d 6f 64 69 66 69 65 64 3e 31 33 32 38 34 38 32   modified>1328482
0140  39 33 38 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63   938</modified><c
0150  6f 6d 70 6c 65 74 65 64 3e 31 33 32 37 39 32 34   ompleted>1327924
0160  38 30 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c   800</completed><
0170  66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72   folder>0</folder
0180  3e 3c 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e   ><context>0</con
0190  74 65 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 6f   text><goal>0</go
01a0  61 6c 3e 3c 70 72 69 6f 72 69 74 79 3e 30 3c 2f   al><priority>0</
01b0  70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 74   priority><repeat
01c0  3e 3c 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61   ></repeat><repea
01d0  74 66 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 66   tfrom>0</repeatf
01e0  72 6f 6d 3e 3c 64 75 65 64 61 74 65 3e 31 33 32   rom><duedate>132
01f0  38 30 39 37 36 30 30 3c 2f 64 75 65 64 61 74 65   8097600</duedate
0200  3e 3c 73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73   ><startdate>0</s
0210  74 61 72 74 64 61 74 65 3e 3c 64 75 65 74 69 6d   tartdate><duetim
0220  65 3e 31 33 32 38 31 33 37 32 30 30 3c 2f 64 75   e>1328137200</du
0230  65 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 6d 65   etime><starttime
0240  3e 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c   >0</starttime><l
0250  65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e   ength>0</length>
0260  3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75   <status>2</statu
0270  73 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c   s><note></note><
0280  2f 74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e   /task><task><id>
0290  39 31 30 38 33 30 39 3c 2f 69 64 3e 3c 74 69 74   9108309</id><tit
02a0  6c 65 3e 54 65 73 74 20 32 20 2d 66 6f 6f 62 61   le>Test 2 -fooba
02b0  72 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69   r</title><modifi
02c0  65 64 3e 31 33 32 38 37 39 30 38 38 32 3c 2f 6d   ed>1328790882</m
02d0  6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70 6c 65 74   odified><complet
02e0  65 64 3e 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e   ed>0</completed>
02f0  3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65   <folder>0</folde
0300  72 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f   r><context>0</co
0310  6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67   ntext><goal>0</g
0320  6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79 3e 30 3c   oal><priority>0<
0330  2f 70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61   /priority><repea
0340  74 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 65 70 65   t></repeat><repe
0350  61 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74   atfrom>0</repeat
0360  66 72 6f 6d 3e 3c 64 75 65 64 61 74 65 3e 30 3c   from><duedate>0<
0370  2f 64 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64   /duedate><startd
0380  61 74 65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65   ate>0</startdate
0390  3e 3c 64 75 65 74 69 6d 65 3e 30 3c 2f 64 75 65   ><duetime>0</due
03a0  74 69 6d 65 3e 3c 73 74 61 72 74 74 69 6d 65 3e   time><starttime>
03b0  30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c 65   0</starttime><le
03c0  6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e 3c   ngth>0</length><
03d0  73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 73   status>2</status
03e0  3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c 2f   ><note></note></
03f0  74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e 39   task><task><id>9
0400  31 34 39 33 33 33 3c 2f 69 64 3e 3c 74 69 74 6c   149333</id><titl
0410  65 3e 6f 72 67 2d 74 6f 6f 64 6c 65 64 6f 3c 2f   e>org-toodledo</
0420  74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e   title><modified>
0430  31 33 32 38 30 31 31 39 34 39 3c 2f 6d 6f 64 69   1328011949</modi
0440  66 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e   fied><completed>
0450  30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 66 6f   0</completed><fo
0460  6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72 3e 3c   lder>0</folder><
0470  63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e 74 65   context>0</conte
0480  78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 6f 61 6c   xt><goal>0</goal
0490  3e 3c 70 72 69 6f 72 69 74 79 3e 32 3c 2f 70 72   ><priority>2</pr
04a0  69 6f 72 69 74 79 3e 3c 72 65 70 65 61 74 3e 3c   iority><repeat><
04b0  2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61 74 66   /repeat><repeatf
04c0  72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 66 72 6f   rom>0</repeatfro
04d0  6d 3e 3c 64 75 65 64 61 74 65 3e 30 3c 2f 64 75   m><duedate>0</du
04e0  65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74 65   edate><startdate
04f0  3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c 64   >0</startdate><d
0500  75 65 74 69 6d 65 3e 30 3c 2f 64 75 65 74 69 6d   uetime>0</duetim
0510  65 3e 3c 73 74 61 72 74 74 69 6d 65 3e 30 3c 2f   e><starttime>0</
0520  73 74 61 72 74 74 69 6d 65 3e 3c 6c 65 6e 67 74   starttime><lengt
0530  68 3e 30 3c 2f 6c 65 6e 67 74 68 3e 3c 73 74 61   h>0</length><sta
0540  74 75 73 3e 32 3c 2f 73 74 61 74 75 73 3e 3c 6e   tus>2</status><n
0550  6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c 2f 74 61 73   ote></note></tas
0560  6b 3e 3c 2f 74 61 73 6b 73 3e                     k></tasks>
----------------------------------------

And the URL-DEBUG:

----------------------------------------
34:49.154000 http -> Finding free connection: api.toodledo.com:80
34:49.154000 http -> Contacting host: api.toodledo.com:80
34:49.200000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
34:49.201000 http -> Request is: 
POST /2/tasks/get.php HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.toodledo.com
Accept-charset: nil
Accept: */*
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)
Content-Type: application/x-www-form-urlencoded
Content-length: 212

f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0

34:49.201000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer  *http api.toodledo.com:80*<92>>)
34:49.264000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
34:49.264000 http ->    data (point-max 1) (len 1380):
"HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 12:34:49 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1189
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml

<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328790882</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status>
 <note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</!
 priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
34:49.264000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<92>)
34:49.264000 http -> Saw end of headers... ( *http api.toodledo.com:80*<92>) @ 197
34:49.264000 http -> url-http-clean-headers, end-of-headers now at 189
34:49.264000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>)
34:49.264000 http -> Got a content-length 1189, being smart about document end.
34:49.264000 http -> Calling initial content-length for extra data at end of headers
34:49.264000 http -> url-http-content-length-after-change-function 189 1381 1192
34:49.265000 http -> End of document found!
34:49.265000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
34:49.265000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<92>)
34:49.265000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>)
34:49.266000 http -> Parsed HTTP headers: class=2 status=200
34:49.266000 http -> Finished parsing HTTP headers: t
34:49.266000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
34:49.266000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<92>)
----------------------------------------

...cj





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09 12:49   ` Christopher J. White
@ 2012-02-09 13:07     ` Andreas Schwab
  0 siblings, 0 replies; 8+ messages in thread
From: Andreas Schwab @ 2012-02-09 13:07 UTC (permalink / raw)
  To: Christopher J. White; +Cc: 10768

"Christopher J. White" <chris@grierwhite.com> writes:

> I considered that and looked hard for it to be the server.  In the data
> output the double quote chars add an additional slash "\" that is not
> present in the actual data.

You are right, I missed that.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09  4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White
  2012-02-09  9:15 ` Andreas Schwab
@ 2012-02-09 15:43 ` Andreas Schwab
  2012-02-09 18:46   ` Christopher J. White
  2012-02-09 20:14   ` Christopher J. White
  1 sibling, 2 replies; 8+ messages in thread
From: Andreas Schwab @ 2012-02-09 15:43 UTC (permalink / raw)
  To: Christopher J. White; +Cc: 10768

Does this help?

diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index b43ed76..140824f 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -352,11 +352,14 @@ request.")
 ;; Parsing routines
 (defun url-http-clean-headers ()
   "Remove trailing \r from header lines.
-This allows us to use `mail-fetch-field', etc."
+This allows us to use `mail-fetch-field', etc.
+Return the number of characters removed."
   (declare (special url-http-end-of-headers))
-  (goto-char (point-min))
-  (while (re-search-forward "\r$" url-http-end-of-headers t)
-    (replace-match "")))
+  (let ((end (marker-position url-http-end-of-headers)))
+    (goto-char (point-min))
+    (while (re-search-forward "\r$" url-http-end-of-headers t)
+      (replace-match ""))
+    (- end url-http-end-of-headers)))
 
 (defun url-http-handle-authentication (proxy)
   (declare (special status success url-http-method url-http-data
@@ -1051,7 +1054,7 @@ the end of the document."
 	  (setq url-http-end-of-headers (set-marker (make-marker)
 						    (point))
 		end-of-headers t)
-	  (url-http-clean-headers)))
+	  (setq nd (- nd (url-http-clean-headers))))
 
       (if (not end-of-headers)
 	  ;; Haven't seen the end of the headers yet, need to wait

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."





^ permalink raw reply related	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09 15:43 ` Andreas Schwab
@ 2012-02-09 18:46   ` Christopher J. White
  2012-02-09 20:14   ` Christopher J. White
  1 sibling, 0 replies; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 18:46 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: 10768

Quick work!  First past looks good, except you're missing one additional
paren after the last diff line:

-	  (url-http-clean-headers)))
+	  (setq nd (- nd (url-http-clean-headers)))))

Let me run this through a number of cases and see how well it holds up
for a few days.

Thanks
...cj


On Thu, 09 Feb 2012 16:43:27 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:

> Does this help?
> 
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index b43ed76..140824f 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -352,11 +352,14 @@ request.")
>  ;; Parsing routines
>  (defun url-http-clean-headers ()
>    "Remove trailing \r from header lines.
> -This allows us to use `mail-fetch-field', etc."
> +This allows us to use `mail-fetch-field', etc.
> +Return the number of characters removed."
>    (declare (special url-http-end-of-headers))
> -  (goto-char (point-min))
> -  (while (re-search-forward "\r$" url-http-end-of-headers t)
> -    (replace-match "")))
> +  (let ((end (marker-position url-http-end-of-headers)))
> +    (goto-char (point-min))
> +    (while (re-search-forward "\r$" url-http-end-of-headers t)
> +      (replace-match ""))
> +    (- end url-http-end-of-headers)))
>  
>  (defun url-http-handle-authentication (proxy)
>    (declare (special status success url-http-method url-http-data
> @@ -1051,7 +1054,7 @@ the end of the document."
>  	  (setq url-http-end-of-headers (set-marker (make-marker)
>  						    (point))
>  		end-of-headers t)
> -	  (url-http-clean-headers)))
> +	  (setq nd (- nd (url-http-clean-headers))))
>  
>        (if (not end-of-headers)
>  	  ;; Haven't seen the end of the headers yet, need to wait
> 
> Andreas.
> 






^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09 15:43 ` Andreas Schwab
  2012-02-09 18:46   ` Christopher J. White
@ 2012-02-09 20:14   ` Christopher J. White
  2012-02-10 11:47     ` Andreas Schwab
  1 sibling, 1 reply; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 20:14 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: 10768

Hi Andreas,

I have created a small set of files with lengths varying from
1200 to 1225, and from 2658 to 2662 and put them up on my personal web
server.  (e.g. http://www.grierwhite.com/url-http-1220.txt)

I chose lengths that just barely stretch from 1 to 2 payload packets,
and from 2 to 3 packets. 

1200-1207 -- 1 packets
1208-1225 -- 2 packets, 1-18 bytes in the 2nd
2658-2659 -- 2 packets
2660-2662 -- 3 packets, 1-3 bytes in the 3rd

Prior to the change, the results are as follows:

1200-1207 -- pass
1208-1217 -- fail
1217-1225 -- pass
2658-2662 -- pass

This indicates that the failure occurs when there are up to 10 bytes
in the 2nd packet.  This correlates with the fact that the header is 10
lines, thus will replace 10 \r characters and shorten the buffer by
that amount.  Consistent with the results I found with the original
server toodledo.com that only has 8 header lines and only has the issue
up to 8 bytes in the 2nd packet.

It is interesting to note that when there are 3 packets and just a few
bytes in the last packet, the failure does *not* occur.  I think this
may be explained by the fact that 
url-http-wait-for-headers-change-function may be called a second time,
thus all position references are recomputed.  Andreas, can you verify
this logic?  If this is correct, the problem only occurs for responses
that just border on 2 packets.

After the change, all lengths pass.  

Here is the lisp code that tests the lengths, feel free to run this as
you should be able to reproduce this from any machine that can access
the internet.  I don't have a trunk version of emacs (24), but looking
at the trunk copy of url-http.el, it appears to suffer from the same
problem.

(defun test-url-http-patch (minlen maxlen)
  (let ((passed 0) (failed 0))
    (do ((len minlen (+ 1 len))) ((> len maxlen) nil)
      (set-buffer 
       (url-retrieve-synchronously 
        (format "http://www.grierwhite.com/url-http-%d.txt" len)))
      (goto-char (point-min))
      (re-search-forward "^\n")
      (let ((recvlen (- (point-max) (point))))
        (if (= len recvlen)
            (setq passed (1+ passed))
          (setq failed (1+ failed))
          (message "FAILED orig-len: %d, received len: %d" 
                   len recvlen))))
    (message "Passed: %d/%d" passed (+ passed failed))))

;; Run without the patch
(test-url-http-patch 1200 1225)
"Passed: 16/26"

(test-url-http-patch 2658 2662)
"Passed: 5/5"

;; Run with the patch
(test-url-http-patch 1200 1225)
"Passed: 26/26"

(test-url-http-patch 2658 2662)
"Passed: 5/5"

...cj

On Thu, 09 Feb 2012 16:43:27 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:

> Does this help?
> 
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index b43ed76..140824f 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -352,11 +352,14 @@ request.")
>  ;; Parsing routines
>  (defun url-http-clean-headers ()
>    "Remove trailing \r from header lines.
> -This allows us to use `mail-fetch-field', etc."
> +This allows us to use `mail-fetch-field', etc.
> +Return the number of characters removed."
>    (declare (special url-http-end-of-headers))
> -  (goto-char (point-min))
> -  (while (re-search-forward "\r$" url-http-end-of-headers t)
> -    (replace-match "")))
> +  (let ((end (marker-position url-http-end-of-headers)))
> +    (goto-char (point-min))
> +    (while (re-search-forward "\r$" url-http-end-of-headers t)
> +      (replace-match ""))
> +    (- end url-http-end-of-headers)))
>  
>  (defun url-http-handle-authentication (proxy)
>    (declare (special status success url-http-method url-http-data
> @@ -1051,7 +1054,7 @@ the end of the document."
>  	  (setq url-http-end-of-headers (set-marker (make-marker)
>  						    (point))
>  		end-of-headers t)
> -	  (url-http-clean-headers)))
> +	  (setq nd (- nd (url-http-clean-headers))))
>  
>        (if (not end-of-headers)
>  	  ;; Haven't seen the end of the headers yet, need to wait
> 
> Andreas.
> 






^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
  2012-02-09 20:14   ` Christopher J. White
@ 2012-02-10 11:47     ` Andreas Schwab
  0 siblings, 0 replies; 8+ messages in thread
From: Andreas Schwab @ 2012-02-10 11:47 UTC (permalink / raw)
  To: Christopher J. White; +Cc: 10768-done

Thanks for testing, installed in trunk.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."





^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2012-02-10 11:47 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-02-09  4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White
2012-02-09  9:15 ` Andreas Schwab
2012-02-09 12:49   ` Christopher J. White
2012-02-09 13:07     ` Andreas Schwab
2012-02-09 15:43 ` Andreas Schwab
2012-02-09 18:46   ` Christopher J. White
2012-02-09 20:14   ` Christopher J. White
2012-02-10 11:47     ` Andreas Schwab

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).