* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
@ 2012-02-09 4:53 Christopher J. White
2012-02-09 9:15 ` Andreas Schwab
2012-02-09 15:43 ` Andreas Schwab
0 siblings, 2 replies; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 4:53 UTC (permalink / raw)
To: 10768
I’m using url-http to retrieve an XML response. In noticed somewhat by
chance that certain page sizes were cutting off the last few characters
of the response, which in turn caused the XML to be poorly formed.
I've noticed the issue occurs when the payload response has to be sent
in two TCP packets and the last packet has 8 or fewer bytes. I observe
this by looking at the HTTP stream in Wireshark.
I spent time tracing the response by adding some debug using
url-http-debug. The first chunk of data comes from the first TCP packet
and includes the HTTP header and all the XML data minus the last 8 bytes.
url-http-wait-for-headers-change-function is called to process this
chunk of data. After processing this chunk, the connection is closed
and the buffer of XML data returned minux the 8 bytes.
Here's the http trace, along with my added debug messages (inlcuding
dumping of the full data):
----------------------------------------
22:06.781000 http -> Contacting host: api.toodledo.com:80
22:06.827000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.828000 http -> Request is:
POST /2/tasks/get.php HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.toodledo.com
Accept-charset: nil
Accept: */*
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)
Content-Type: application/x-www-form-urlencoded
Content-length: 212
f=xml&key=9f2a92de083e8d63c2bd14771661139f&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
22:06.828000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<80>>)
22:06.889000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
22:06.889000 http -> data (point-max 1) (len 1380):
"HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 04:22:06 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1189
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
22:06.889000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<80>)
22:06.890000 http -> Saw end of headers... ( *http api.toodledo.com:80*<80>) @ 197
22:06.890000 http -> url-http-clean-headers, end-of-headers now at 189
22:06.890000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<80>)
22:06.890000 http -> Got a content-length 1189, being smart about document end.
22:06.890000 http -> Calling initial content-length for extra data at end of headers
22:06.890000 http -> url-http-content-length-after-change-function 189 1381 1192
22:06.891000 http -> End of document found!
22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.891000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<80>)
22:06.891000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<80>)
22:06.891000 http -> Parsed HTTP headers: class=2 status=200
22:06.891000 http -> Finished parsing HTTP headers: t
22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.891000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<80>)
----------------------------------------
Notice that the XML in the data finishes with "...</note></task></". It
should finish with: "...</note></task></tasks>".
A key debug message I added is "End of document found!", and this is
placed in the function url-http-content-length-after-change-function
right after the comment "Found the end of the document! Wheee!":
(defun url-http-content-length-after-change-function (st nd length)
<....>
(if (> (- nd url-http-end-of-headers) url-http-content-length)
(progn
;; Found the end of the document! Wheee!
(url-http-debug "End of document found!")
(url-display-percentage nil nil)
(url-lazy-message "Reading... done.")
(if (url-http-parse-headers)
(url-http-activate-callback)))))
If I slightly increase or decrease the amount of data returned
by the XML (by adding or deleting characters from title), I can get it
to work properly or not.
For example, by adding 5 chars, I get:
----------------------------------------
39:20.144000 http -> Finding free connection: api.toodledo.com:80
39:20.144000 http -> Contacting host: api.toodledo.com:80
39:20.187000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.188000 http -> Request is:
POST /2/tasks/get.php HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.toodledo.com
Accept-charset: nil
Accept: */*
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)
Content-Type: application/x-www-form-urlencoded
Content-length: 212
f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
39:20.188000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<86>>)
39:20.242000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
39:20.243000 http -> data (point-max 1) (len 1380):
"HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 04:39:20 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1194
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar12345</title><modified>1328762353</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></ta"
39:20.243000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<86>)
39:20.243000 http -> Saw end of headers... ( *http api.toodledo.com:80*<86>) @ 197
39:20.243000 http -> url-http-clean-headers, end-of-headers now at 189
39:20.243000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<86>)
39:20.243000 http -> Got a content-length 1194, being smart about document end.
39:20.243000 http -> Calling initial content-length for extra data at end of headers
39:20.243000 http -> url-http-content-length-after-change-function 189 1381 1192
39:20.244000 http -> Spinning waiting for headers...
39:20.244000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<86>>)
39:20.247000 http -> Calling after change function `url-http-content-length-after-change-function' for `#<process api.toodledo.com<1>>'
39:20.247000 http -> data (point-max 1373) (len 11):
"sk></tasks>"
39:20.247000 http -> url-http-content-length-after-change-function 1373 1384 11
39:20.247000 http -> End of document found!
39:20.247000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.247000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<86>)
39:20.247000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<86>)
39:20.247000 http -> Parsed HTTP headers: class=2 status=200
39:20.248000 http -> Finished parsing HTTP headers: t
39:20.248000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.248000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<86>)
----------------------------------------
In this case, url-http waits for the remaining data from the second TCP
packet, which comes in slightly later and is pieced together and forms a
good XML document.
If I reduce the size by an appropriate number of bytes so that the XML
data fits entirely in the first packet it also works.
I think the problem relates to url-http-clean-headers. I have 8-lines
of headers, and this function removes the trailing \r from each of those
lines. I think this is not accounted for when using the content-length
to determine if all the data has been received. After dropping these 8
trailing \r charaters, the buffer positions all change.
I'm not quite able to figure out how the content-length is used to
detect the end of processing, so I thought I'd submit this bug and see
if any other folks can point me in the right direction.
...cj
In GNU Emacs 23.3.1 (i386-mingw-nt6.1.7601)
of 2011-03-10 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 6.1.7601
configured using `configure --with-gcc (4.5) --no-opt --cflags -Ic:/imagesupport/include'
Important settings:
value of $LC_ALL: nil
value of $LC_COLLATE: nil
value of $LC_CTYPE: nil
value of $LC_MESSAGES: nil
value of $LC_MONETARY: nil
value of $LC_NUMERIC: nil
value of $LC_TIME: nil
value of $LANG: C.UTF-8
value of $XMODIFIERS: nil
locale-coding-system: cp1252
default enable-multibyte-characters: t
Major mode: Fundamental
Minor modes in effect:
show-paren-mode: t
display-time-mode: t
tooltip-mode: t
mouse-wheel-mode: t
global-font-lock-mode: t
auto-encryption-mode: t
auto-compression-mode: t
column-number-mode: t
line-number-mode: t
transient-mark-mode: t
Recent input:
- - - - - - - - - - - - - - - - <return> <return> <switch-frame>
<help-echo> <select-window> <backspace> <escape> C-x
<select-window> <help-echo> C-/ C-/ C-/ C-/ SPC C-/
C-/ C-a <tab> <tab> C-x [ C-x ] C-x [ C-/ C-/ C-/ C-x
[ C-SPC C-x ] C-w C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-a
<tab> <S-tab> <tab> C-o s <help-echo> <switch-frame>
C-SPC C-x ] <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <down> C-w <up> <down> <switch-frame>
<switch-frame> <down> <down> C-x 2 <prior> <prior>
<down> <down> <down-mouse-1> <mouse-movement> <mouse-1>
<C-left> C-s C-w C-w C-w C-w C-w C-w C-w <return> C-x
o C-s C-s C-a <help-echo> C-a C-l C-n C-n C-n C-n C-n
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n <down>
<down> <down> <down> C-x o C-p C-n C-n C-n C-n C-n
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-l
C-n C-n C-l C-n C-n C-n C-n C-n C-l C-n C-n C-n C-n
C-l C-x o C-n C-n C-n <select-window> <select-window>
<select-window> <help-echo> <prior> <prior> <next>
<switch-frame> <down-mouse-1> <mouse-movement> <mouse-1>
C-o s <select-window> <help-echo> <switch-frame> <down-mouse-1>
<mouse-movement> <mouse-1> C-x ] <prior> <switch-frame>
<down-mouse-1> <mouse-movement> <mouse-1> <down-mouse-1>
<mouse-1> SPC <escape> C-x <select-window> <help-echo>
<select-window> C-x o C-o s <switch-frame> <down-mouse-1>
<mouse-movement> <mouse-1> C-x ] C-r t a s k s C-r
C-r C-r C-s C-a <next> <switch-frame> <switch-frame>
<switch-frame> <switch-frame> <switch-frame> <switch-frame>
<switch-frame> <switch-frame> <escape> x r e p o <tab>
r t SPC e m SPC SPC <return>
Recent messages:
Reading [application/xml
]... 1k of 1k (100%)
Contacting host: api.toodledo.com:80
Reading [application/xml
]... 718 bytes of 717 bytes (100%)
Sync complete, 794 changes: recv 3 mod 791 del
Mark set
Mark saved where search started
Auto-saving...done
Making completion list...
Load-path shadows:
~cwhite/lib/lisp/org-toodledo/w3mexcerpt hides ~cwhite/lib/lisp/w3mexcerpt
~cwhite/lib/lisp/org-toodledo/http-post-simple hides ~cwhite/lib/lisp/http-post-simple
~/lib/lisp/org-mode/lisp/org hides c:/Users/cwhite/emacs-23.3/lisp/org/org
~/lib/lisp/org-mode/lisp/org-xoxo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-xoxo
~/lib/lisp/org-mode/lisp/org-wl hides c:/Users/cwhite/emacs-23.3/lisp/org/org-wl
~/lib/lisp/org-mode/lisp/org-w3m hides c:/Users/cwhite/emacs-23.3/lisp/org/org-w3m
~/lib/lisp/org-mode/lisp/org-vm hides c:/Users/cwhite/emacs-23.3/lisp/org/org-vm
~/lib/lisp/org-mode/lisp/org-timer hides c:/Users/cwhite/emacs-23.3/lisp/org/org-timer
~/lib/lisp/org-mode/lisp/org-table hides c:/Users/cwhite/emacs-23.3/lisp/org/org-table
~/lib/lisp/org-mode/lisp/org-src hides c:/Users/cwhite/emacs-23.3/lisp/org/org-src
~/lib/lisp/org-mode/lisp/org-rmail hides c:/Users/cwhite/emacs-23.3/lisp/org/org-rmail
~/lib/lisp/org-mode/lisp/org-remember hides c:/Users/cwhite/emacs-23.3/lisp/org/org-remember
~/lib/lisp/org-mode/lisp/org-publish hides c:/Users/cwhite/emacs-23.3/lisp/org/org-publish
~/lib/lisp/org-mode/lisp/org-protocol hides c:/Users/cwhite/emacs-23.3/lisp/org/org-protocol
~/lib/lisp/org-mode/lisp/org-plot hides c:/Users/cwhite/emacs-23.3/lisp/org/org-plot
~/lib/lisp/org-mode/lisp/org-mouse hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mouse
~/lib/lisp/org-mode/lisp/org-mobile hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mobile
~/lib/lisp/org-mode/lisp/org-mhe hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mhe
~/lib/lisp/org-mode/lisp/org-mew hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mew
~/lib/lisp/org-mode/lisp/org-macs hides c:/Users/cwhite/emacs-23.3/lisp/org/org-macs
~/lib/lisp/org-mode/lisp/org-mac-message hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mac-message
~/lib/lisp/org-mode/lisp/org-list hides c:/Users/cwhite/emacs-23.3/lisp/org/org-list
~/lib/lisp/org-mode/lisp/org-latex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-latex
~/lib/lisp/org-mode/lisp/org-jsinfo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-jsinfo
~/lib/lisp/org-mode/lisp/org-irc hides c:/Users/cwhite/emacs-23.3/lisp/org/org-irc
~/lib/lisp/org-mode/lisp/org-install hides c:/Users/cwhite/emacs-23.3/lisp/org/org-install
~/lib/lisp/org-mode/lisp/org-inlinetask hides c:/Users/cwhite/emacs-23.3/lisp/org/org-inlinetask
~/lib/lisp/org-mode/lisp/org-info hides c:/Users/cwhite/emacs-23.3/lisp/org/org-info
~/lib/lisp/org-mode/lisp/org-indent hides c:/Users/cwhite/emacs-23.3/lisp/org/org-indent
~/lib/lisp/org-mode/lisp/org-id hides c:/Users/cwhite/emacs-23.3/lisp/org/org-id
~/lib/lisp/org-mode/lisp/org-icalendar hides c:/Users/cwhite/emacs-23.3/lisp/org/org-icalendar
~/lib/lisp/org-mode/lisp/org-html hides c:/Users/cwhite/emacs-23.3/lisp/org/org-html
~/lib/lisp/org-mode/lisp/org-habit hides c:/Users/cwhite/emacs-23.3/lisp/org/org-habit
~/lib/lisp/org-mode/lisp/org-gnus hides c:/Users/cwhite/emacs-23.3/lisp/org/org-gnus
~/lib/lisp/org-mode/lisp/org-freemind hides c:/Users/cwhite/emacs-23.3/lisp/org/org-freemind
~/lib/lisp/org-mode/lisp/org-footnote hides c:/Users/cwhite/emacs-23.3/lisp/org/org-footnote
~/lib/lisp/org-mode/lisp/org-feed hides c:/Users/cwhite/emacs-23.3/lisp/org/org-feed
~/lib/lisp/org-mode/lisp/org-faces hides c:/Users/cwhite/emacs-23.3/lisp/org/org-faces
~/lib/lisp/org-mode/lisp/org-exp hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp
~/lib/lisp/org-mode/lisp/org-exp-blocks hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp-blocks
~/lib/lisp/org-mode/lisp/org-docbook hides c:/Users/cwhite/emacs-23.3/lisp/org/org-docbook
~/lib/lisp/org-mode/lisp/org-datetree hides c:/Users/cwhite/emacs-23.3/lisp/org/org-datetree
~/lib/lisp/org-mode/lisp/org-crypt hides c:/Users/cwhite/emacs-23.3/lisp/org/org-crypt
~/lib/lisp/org-mode/lisp/org-compat hides c:/Users/cwhite/emacs-23.3/lisp/org/org-compat
~/lib/lisp/org-mode/lisp/org-colview hides c:/Users/cwhite/emacs-23.3/lisp/org/org-colview
~/lib/lisp/org-mode/lisp/org-clock hides c:/Users/cwhite/emacs-23.3/lisp/org/org-clock
~/lib/lisp/org-mode/lisp/org-bibtex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bibtex
~/lib/lisp/org-mode/lisp/org-bbdb hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bbdb
~/lib/lisp/org-mode/lisp/org-attach hides c:/Users/cwhite/emacs-23.3/lisp/org/org-attach
~/lib/lisp/org-mode/lisp/org-ascii hides c:/Users/cwhite/emacs-23.3/lisp/org/org-ascii
~/lib/lisp/org-mode/lisp/org-archive hides c:/Users/cwhite/emacs-23.3/lisp/org/org-archive
~/lib/lisp/org-mode/lisp/org-agenda hides c:/Users/cwhite/emacs-23.3/lisp/org/org-agenda
Features:
(shadow sort mail-extr message ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailabbrev nnheader
gnus-util netrc gmm-utils mailheader canlock sha1 hex-util hashcash
emacsbug sgml-mode cl-specs edebug find-func vc-git nxml-uchnm rng-xsd
xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok browse-url
cperl-mode url-cache org-colview diary-lib diary-loaddefs cal-iso
org-table bookmark pp image-file mule-util grep sh-script executable
tabify org-ascii help-mode view org-wl org-w3m org-vm org-rmail org-mhe
org-mew org-irc org-jsinfo org-infojs org-html format-spec org-exp
ob-exp org-exp-blocks org-agenda org-info org-gnus org-docview
org-bibtex bibtex org-bbdb multi-isearch newcomment org-toodledo
http-post-simple url-http tls url-auth mail-parse rfc2231 rfc2047
rfc2045 qp ietf-drums url-gw url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-util url-parse url-vars mm-util
mail-prsvr mailcap json xml w3mexcerpt remember org-remember
org-datetree org ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table
org-footnote org-src ob-comint ob-keys ob ob-eval org-pcomplete
pcomplete org-list org-faces org-compat org-entities org-macs time-date
noutline outline easy-mmode cal-menu calendar cal-loaddefs cygwin-mount
ange-ftp uniquify filecache server byte-opt warnings advice help-fns
advice-preload paren time etags python-mode info-look css-mode scr rsync
twiki edmacro kmacro info doku pib php-mode speedbar sb-image ezimage
dframe assoc highline pop3 mail-utils cl cl-19 cc-mode cc-fonts cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs regexp-opt cus-edit
easymenu cus-start cus-load wid-edit exec compile comint ring bytecomp
byte-compile tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32
disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty emacs)
^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White
@ 2012-02-09 9:15 ` Andreas Schwab
2012-02-09 12:49 ` Christopher J. White
2012-02-09 15:43 ` Andreas Schwab
1 sibling, 1 reply; 8+ messages in thread
From: Andreas Schwab @ 2012-02-09 9:15 UTC (permalink / raw)
To: Christopher J. White; +Cc: 10768
"Christopher J. White" <chris@grierwhite.com> writes:
> "HTTP/1.1 200 OK
> Date: Thu, 09 Feb 2012 04:22:06 GMT
> Server: Apache/2.2.3 (Red Hat)
> Content-Length: 1189
^^^^
> Keep-Alive: timeout=2, max=100
> Connection: Keep-Alive
> Content-Type: application/xml
>
> <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
1189 bytes of data follow.
That looks like a bug in the server.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 9:15 ` Andreas Schwab
@ 2012-02-09 12:49 ` Christopher J. White
2012-02-09 13:07 ` Andreas Schwab
0 siblings, 1 reply; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 12:49 UTC (permalink / raw)
To: Andreas Schwab; +Cc: 10768
On Thu, 09 Feb 2012 10:15:47 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:
> "Christopher J. White" <chris@grierwhite.com> writes:
> > Content-Length: 1189
> ^^^^
> > <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3'
> > total='3'><task><id>9105869</id><title>Test
> > 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test
> > 2
> > -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
>
> 1189 bytes of data follow.
>
> That looks like a bug in the server.
I considered that and looked hard for it to be the server. In the data
output the double quote chars add an additional slash "\" that is not
present in the actual data. The content-length and the actual number
of raw bytes matches precisely when decoded from wireshark.
I reran the test and collected the 3 sets of data, the recombined HTTM
response as built by wireshark (TCP Follow Stream), the ascii dump
of the wireshark packets (2 involved), and the URL-Debug. The key point
is that the data in the second packet is never processed by url-http, only
the bytes in the first packet.
HTTP response as seen in Wireshark:
----------------------------------------
f=xml&key=444ed11d926131624b2ce2ed81f58ad4&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 12:34:49 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1189
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml
<?xml version="1.0" encoding="UTF-8"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328790882</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><not
e></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</prio!
rity><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></tasks>
----------------------------------------
There is no newline at the end, the last character is the ">" after tasks.
The xml data is all a single line, there are no CR/LF characters any where
in it, so if any show up in this message they should be removed if you're
trying to count it out.
Here's the wireshark decode for the 2 packets involved:
----------------------------------------
No. Time Source Destination Protocol Length Info
312 49.870066 72.4.112.215 11.1.1.89 TCP 1434 [TCP segment of a reassembled PDU]
Frame 312: 1434 bytes on wire (11472 bits), 1434 bytes captured (11472 bits)
Arrival Time: Feb 9, 2012 07:34:49.261829000 Eastern Standard Time
Epoch Time: 1328790889.261829000 seconds
[Time delta from previous captured frame: 0.008369000 seconds]
[Time delta from previous displayed frame: 0.008369000 seconds]
[Time since reference or first frame: 49.870066000 seconds]
Frame Number: 312
Frame Length: 1434 bytes (11472 bits)
Capture Length: 1434 bytes (11472 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ip:tcp]
[Coloring Rule Name: HTTP]
[Coloring Rule String: http || tcp.port == 80]
Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 1420
Identification: 0xe548 (58696)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 55
Protocol: TCP (6)
Header checksum: 0x93ee [correct]
[Good: True]
[Bad: False]
Source: 72.4.112.215 (72.4.112.215)
Destination: 11.1.1.89 (11.1.1.89)
Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1, Ack: 527, Len: 1380
Source port: http (80)
Destination port: 60823 (60823)
[Stream index: 10]
Sequence number: 1 (relative sequence number)
[Next sequence number: 1381 (relative sequence number)]
Acknowledgement number: 527 (relative ack number)
Header length: 20 bytes
Flags: 0x10 (ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgement: Set
.... .... 0... = Push: Not set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
Window size value: 54
[Calculated window size: 6912]
[Window size scaling factor: 128]
Checksum: 0x836f [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
[SEQ/ACK analysis]
[Bytes in flight: 1380]
TCP segment data (1380 bytes)
0000 00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00 .!k^.....D....E.
0010 05 8c e5 48 40 00 37 06 93 ee 48 04 70 d7 0b 01 ...H@.7...H.p...
0020 01 59 00 50 ed 97 30 cd 5f f0 eb ef 69 24 50 10 .Y.P..0._...i$P.
0030 00 36 83 6f 00 00 48 54 54 50 2f 31 2e 31 20 32 .6.o..HTTP/1.1 2
0040 30 30 20 4f 4b 0d 0a 44 61 74 65 3a 20 54 68 75 00 OK..Date: Thu
0050 2c 20 30 39 20 46 65 62 20 32 30 31 32 20 31 32 , 09 Feb 2012 12
0060 3a 33 34 3a 34 39 20 47 4d 54 0d 0a 53 65 72 76 :34:49 GMT..Serv
0070 65 72 3a 20 41 70 61 63 68 65 2f 32 2e 32 2e 33 er: Apache/2.2.3
0080 20 28 52 65 64 20 48 61 74 29 0d 0a 43 6f 6e 74 (Red Hat)..Cont
0090 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 31 38 39 ent-Length: 1189
00a0 0d 0a 4b 65 65 70 2d 41 6c 69 76 65 3a 20 74 69 ..Keep-Alive: ti
00b0 6d 65 6f 75 74 3d 32 2c 20 6d 61 78 3d 31 30 30 meout=2, max=100
00c0 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 65 ..Connection: Ke
00d0 65 70 2d 41 6c 69 76 65 0d 0a 43 6f 6e 74 65 6e ep-Alive..Conten
00e0 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat
00f0 69 6f 6e 2f 78 6d 6c 0d 0a 0d 0a 3c 3f 78 6d 6c ion/xml....<?xml
0100 20 76 65 72 73 69 6f 6e 3d 22 31 2e 30 22 20 65 version="1.0" e
0110 6e 63 6f 64 69 6e 67 3d 22 55 54 46 2d 38 22 3f ncoding="UTF-8"?
0120 3e 3c 74 61 73 6b 73 20 6e 75 6d 3d 27 33 27 20 ><tasks num='3'
0130 74 6f 74 61 6c 3d 27 33 27 3e 3c 74 61 73 6b 3e total='3'><task>
0140 3c 69 64 3e 39 31 30 35 38 36 39 3c 2f 69 64 3e <id>9105869</id>
0150 3c 74 69 74 6c 65 3e 54 65 73 74 20 31 3c 2f 74 <title>Test 1</t
0160 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e 31 itle><modified>1
0170 33 32 38 34 38 32 39 33 38 3c 2f 6d 6f 64 69 66 328482938</modif
0180 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e 31 ied><completed>1
0190 33 32 37 39 32 34 38 30 30 3c 2f 63 6f 6d 70 6c 327924800</compl
01a0 65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f eted><folder>0</
01b0 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e folder><context>
01c0 30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 0</context><goal
01d0 3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 >0</goal><priori
01e0 74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e 3c ty>0</priority><
01f0 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e repeat></repeat>
0200 3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72 <repeatfrom>0</r
0210 65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61 epeatfrom><dueda
0220 74 65 3e 31 33 32 38 30 39 37 36 30 30 3c 2f 64 te>1328097600</d
0230 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74 uedate><startdat
0240 65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c e>0</startdate><
0250 64 75 65 74 69 6d 65 3e 31 33 32 38 31 33 37 32 duetime>13281372
0260 30 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 00</duetime><sta
0270 72 74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74 rttime>0</startt
0280 69 6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c ime><length>0</l
0290 65 6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c ength><status>2<
02a0 2f 73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f /status><note></
02b0 6e 6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73 note></task><tas
02c0 6b 3e 3c 69 64 3e 39 31 30 38 33 30 39 3c 2f 69 k><id>9108309</i
02d0 64 3e 3c 74 69 74 6c 65 3e 54 65 73 74 20 32 20 d><title>Test 2
02e0 2d 66 6f 6f 62 61 72 3c 2f 74 69 74 6c 65 3e 3c -foobar</title><
02f0 6d 6f 64 69 66 69 65 64 3e 31 33 32 38 37 39 30 modified>1328790
0300 38 38 32 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 882</modified><c
0310 6f 6d 70 6c 65 74 65 64 3e 30 3c 2f 63 6f 6d 70 ompleted>0</comp
0320 6c 65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c leted><folder>0<
0330 2f 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 /folder><context
0340 3e 30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 >0</context><goa
0350 6c 3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 l>0</goal><prior
0360 69 74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e ity>0</priority>
0370 3c 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 <repeat></repeat
0380 3e 3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f ><repeatfrom>0</
0390 72 65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 repeatfrom><dued
03a0 61 74 65 3e 30 3c 2f 64 75 65 64 61 74 65 3e 3c ate>0</duedate><
03b0 73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73 74 61 startdate>0</sta
03c0 72 74 64 61 74 65 3e 3c 64 75 65 74 69 6d 65 3e rtdate><duetime>
03d0 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 72 0</duetime><star
03e0 74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74 69 ttime>0</startti
03f0 6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65 me><length>0</le
0400 6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c 2f ngth><status>2</
0410 73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e status><note></n
0420 6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73 6b ote></task><task
0430 3e 3c 69 64 3e 39 31 34 39 33 33 33 3c 2f 69 64 ><id>9149333</id
0440 3e 3c 74 69 74 6c 65 3e 6f 72 67 2d 74 6f 6f 64 ><title>org-tood
0450 6c 65 64 6f 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 ledo</title><mod
0460 69 66 69 65 64 3e 31 33 32 38 30 31 31 39 34 39 ified>1328011949
0470 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70 </modified><comp
0480 6c 65 74 65 64 3e 30 3c 2f 63 6f 6d 70 6c 65 74 leted>0</complet
0490 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f ed><folder>0</fo
04a0 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c lder><context>0<
04b0 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 /context><goal>0
04c0 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79 </goal><priority
04d0 3e 32 3c 2f 70 72 69 6f 72 69 74 79 3e 3c 72 65 >2</priority><re
04e0 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 peat></repeat><r
04f0 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 epeatfrom>0</rep
0500 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61 74 65 eatfrom><duedate
0510 3e 30 3c 2f 64 75 65 64 61 74 65 3e 3c 73 74 61 >0</duedate><sta
0520 72 74 64 61 74 65 3e 30 3c 2f 73 74 61 72 74 64 rtdate>0</startd
0530 61 74 65 3e 3c 64 75 65 74 69 6d 65 3e 30 3c 2f ate><duetime>0</
0540 64 75 65 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 duetime><startti
0550 6d 65 3e 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e me>0</starttime>
0560 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 <length>0</lengt
0570 68 3e 3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 h><status>2</sta
0580 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 tus><note></note
0590 3e 3c 2f 74 61 73 6b 3e 3c 2f ></task></
No. Time Source Destination Protocol Length Info
313 49.874992 72.4.112.215 11.1.1.89 HTTP/XML 60 HTTP/1.1 200 OK
Frame 313: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Arrival Time: Feb 9, 2012 07:34:49.266755000 Eastern Standard Time
Epoch Time: 1328790889.266755000 seconds
[Time delta from previous captured frame: 0.004926000 seconds]
[Time delta from previous displayed frame: 0.004926000 seconds]
[Time since reference or first frame: 49.874992000 seconds]
Frame Number: 313
Frame Length: 60 bytes (480 bits)
Capture Length: 60 bytes (480 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ip:tcp:http:xml]
[Coloring Rule Name: HTTP]
[Coloring Rule String: http || tcp.port == 80]
Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 46
Identification: 0xe549 (58697)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 55
Protocol: TCP (6)
Header checksum: 0x994b [correct]
[Good: True]
[Bad: False]
Source: 72.4.112.215 (72.4.112.215)
Destination: 11.1.1.89 (11.1.1.89)
Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1381, Ack: 527, Len: 6
Source port: http (80)
Destination port: 60823 (60823)
[Stream index: 10]
Sequence number: 1381 (relative sequence number)
[Next sequence number: 1387 (relative sequence number)]
Acknowledgement number: 527 (relative ack number)
Header length: 20 bytes
Flags: 0x18 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgement: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
Window size value: 54
[Calculated window size: 6912]
[Window size scaling factor: 128]
Checksum: 0xb632 [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
[SEQ/ACK analysis]
[Bytes in flight: 1386]
TCP segment data (6 bytes)
[2 Reassembled TCP Segments (1386 bytes): #312(1380), #313(6)]
[Frame: 312, payload: 0-1379 (1380 bytes)]
[Frame: 313, payload: 1380-1385 (6 bytes)]
[Segment count: 2]
[Reassembled TCP length: 1386]
Hypertext Transfer Protocol
HTTP/1.1 200 OK\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
[Message: HTTP/1.1 200 OK\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Version: HTTP/1.1
Status Code: 200
Response Phrase: OK
Date: Thu, 09 Feb 2012 12:34:49 GMT\r\n
Server: Apache/2.2.3 (Red Hat)\r\n
Content-Length: 1189\r\n
[Content length: 1189]
Keep-Alive: timeout=2, max=100\r\n
Connection: Keep-Alive\r\n
Content-Type: application/xml\r\n
\r\n
eXtensible Markup Language
<?xml
version="1.0"
encoding="UTF-8"
?>
<tasks
num='3'
total='3'>
<task>
<id>
9105869
</id>
<title>
Test 1
</title>
<modified>
1328482938
</modified>
<completed>
1327924800
</completed>
<folder>
0
</folder>
<context>
0
</context>
<goal>
0
</goal>
<priority>
0
</priority>
<repeat>
</repeat>
<repeatfrom>
0
</repeatfrom>
<duedate>
1328097600
</duedate>
<startdate>
0
</startdate>
<duetime>
1328137200
</duetime>
<starttime>
0
</starttime>
<length>
0
</length>
<status>
2
</status>
<note>
</note>
</task>
<task>
<id>
9108309
</id>
<title>
Test 2 -foobar
</title>
<modified>
1328790882
</modified>
<completed>
0
</completed>
<folder>
0
</folder>
<context>
0
</context>
<goal>
0
</goal>
<priority>
0
</priority>
<repeat>
</repeat>
<repeatfrom>
0
</repeatfrom>
<duedate>
0
</duedate>
<startdate>
0
</startdate>
<duetime>
0
</duetime>
<starttime>
0
</starttime>
<length>
0
</length>
<status>
2
</status>
<note>
</note>
</task>
<task>
<id>
9149333
</id>
<title>
org-toodledo
</title>
<modified>
1328011949
</modified>
<completed>
0
</completed>
<folder>
0
</folder>
<context>
0
</context>
<goal>
0
</goal>
<priority>
2
</priority>
<repeat>
</repeat>
<repeatfrom>
0
</repeatfrom>
<duedate>
0
</duedate>
<startdate>
0
</startdate>
<duetime>
0
</duetime>
<starttime>
0
</starttime>
<length>
0
</length>
<status>
2
</status>
<note>
</note>
</task>
</tasks>
Frame (60 bytes):
0000 00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00 .!k^.....D....E.
0010 00 2e e5 49 40 00 37 06 99 4b 48 04 70 d7 0b 01 ...I@.7..KH.p...
0020 01 59 00 50 ed 97 30 cd 65 54 eb ef 69 24 50 18 .Y.P..0.eT..i$P.
0030 00 36 b6 32 00 00 74 61 73 6b 73 3e .6.2..tasks>
Reassembled TCP (1386 bytes):
0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010 0a 44 61 74 65 3a 20 54 68 75 2c 20 30 39 20 46 .Date: Thu, 09 F
0020 65 62 20 32 30 31 32 20 31 32 3a 33 34 3a 34 39 eb 2012 12:34:49
0030 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 GMT..Server: Ap
0040 61 63 68 65 2f 32 2e 32 2e 33 20 28 52 65 64 20 ache/2.2.3 (Red
0050 48 61 74 29 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 Hat)..Content-Le
0060 6e 67 74 68 3a 20 31 31 38 39 0d 0a 4b 65 65 70 ngth: 1189..Keep
0070 2d 41 6c 69 76 65 3a 20 74 69 6d 65 6f 75 74 3d -Alive: timeout=
0080 32 2c 20 6d 61 78 3d 31 30 30 0d 0a 43 6f 6e 6e 2, max=100..Conn
0090 65 63 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 ection: Keep-Ali
00a0 76 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 ve..Content-Type
00b0 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d : application/xm
00c0 6c 0d 0a 0d 0a 3c 3f 78 6d 6c 20 76 65 72 73 69 l....<?xml versi
00d0 6f 6e 3d 22 31 2e 30 22 20 65 6e 63 6f 64 69 6e on="1.0" encodin
00e0 67 3d 22 55 54 46 2d 38 22 3f 3e 3c 74 61 73 6b g="UTF-8"?><task
00f0 73 20 6e 75 6d 3d 27 33 27 20 74 6f 74 61 6c 3d s num='3' total=
0100 27 33 27 3e 3c 74 61 73 6b 3e 3c 69 64 3e 39 31 '3'><task><id>91
0110 30 35 38 36 39 3c 2f 69 64 3e 3c 74 69 74 6c 65 05869</id><title
0120 3e 54 65 73 74 20 31 3c 2f 74 69 74 6c 65 3e 3c >Test 1</title><
0130 6d 6f 64 69 66 69 65 64 3e 31 33 32 38 34 38 32 modified>1328482
0140 39 33 38 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 938</modified><c
0150 6f 6d 70 6c 65 74 65 64 3e 31 33 32 37 39 32 34 ompleted>1327924
0160 38 30 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 800</completed><
0170 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72 folder>0</folder
0180 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e ><context>0</con
0190 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 6f text><goal>0</go
01a0 61 6c 3e 3c 70 72 69 6f 72 69 74 79 3e 30 3c 2f al><priority>0</
01b0 70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 74 priority><repeat
01c0 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61 ></repeat><repea
01d0 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 66 tfrom>0</repeatf
01e0 72 6f 6d 3e 3c 64 75 65 64 61 74 65 3e 31 33 32 rom><duedate>132
01f0 38 30 39 37 36 30 30 3c 2f 64 75 65 64 61 74 65 8097600</duedate
0200 3e 3c 73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73 ><startdate>0</s
0210 74 61 72 74 64 61 74 65 3e 3c 64 75 65 74 69 6d tartdate><duetim
0220 65 3e 31 33 32 38 31 33 37 32 30 30 3c 2f 64 75 e>1328137200</du
0230 65 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 6d 65 etime><starttime
0240 3e 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c >0</starttime><l
0250 65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e ength>0</length>
0260 3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 <status>2</statu
0270 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c s><note></note><
0280 2f 74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e /task><task><id>
0290 39 31 30 38 33 30 39 3c 2f 69 64 3e 3c 74 69 74 9108309</id><tit
02a0 6c 65 3e 54 65 73 74 20 32 20 2d 66 6f 6f 62 61 le>Test 2 -fooba
02b0 72 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 r</title><modifi
02c0 65 64 3e 31 33 32 38 37 39 30 38 38 32 3c 2f 6d ed>1328790882</m
02d0 6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 odified><complet
02e0 65 64 3e 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e ed>0</completed>
02f0 3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 <folder>0</folde
0300 72 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f r><context>0</co
0310 6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 ntext><goal>0</g
0320 6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79 3e 30 3c oal><priority>0<
0330 2f 70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 /priority><repea
0340 74 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 t></repeat><repe
0350 61 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 atfrom>0</repeat
0360 66 72 6f 6d 3e 3c 64 75 65 64 61 74 65 3e 30 3c from><duedate>0<
0370 2f 64 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 /duedate><startd
0380 61 74 65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 ate>0</startdate
0390 3e 3c 64 75 65 74 69 6d 65 3e 30 3c 2f 64 75 65 ><duetime>0</due
03a0 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 6d 65 3e time><starttime>
03b0 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c 65 0</starttime><le
03c0 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e 3c ngth>0</length><
03d0 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 73 status>2</status
03e0 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c 2f ><note></note></
03f0 74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e 39 task><task><id>9
0400 31 34 39 33 33 33 3c 2f 69 64 3e 3c 74 69 74 6c 149333</id><titl
0410 65 3e 6f 72 67 2d 74 6f 6f 64 6c 65 64 6f 3c 2f e>org-toodledo</
0420 74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e title><modified>
0430 31 33 32 38 30 31 31 39 34 39 3c 2f 6d 6f 64 69 1328011949</modi
0440 66 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e fied><completed>
0450 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 66 6f 0</completed><fo
0460 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72 3e 3c lder>0</folder><
0470 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e 74 65 context>0</conte
0480 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 6f 61 6c xt><goal>0</goal
0490 3e 3c 70 72 69 6f 72 69 74 79 3e 32 3c 2f 70 72 ><priority>2</pr
04a0 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 74 3e 3c iority><repeat><
04b0 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61 74 66 /repeat><repeatf
04c0 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 66 72 6f rom>0</repeatfro
04d0 6d 3e 3c 64 75 65 64 61 74 65 3e 30 3c 2f 64 75 m><duedate>0</du
04e0 65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74 65 edate><startdate
04f0 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c 64 >0</startdate><d
0500 75 65 74 69 6d 65 3e 30 3c 2f 64 75 65 74 69 6d uetime>0</duetim
0510 65 3e 3c 73 74 61 72 74 74 69 6d 65 3e 30 3c 2f e><starttime>0</
0520 73 74 61 72 74 74 69 6d 65 3e 3c 6c 65 6e 67 74 starttime><lengt
0530 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e 3c 73 74 61 h>0</length><sta
0540 74 75 73 3e 32 3c 2f 73 74 61 74 75 73 3e 3c 6e tus>2</status><n
0550 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c 2f 74 61 73 ote></note></tas
0560 6b 3e 3c 2f 74 61 73 6b 73 3e k></tasks>
----------------------------------------
And the URL-DEBUG:
----------------------------------------
34:49.154000 http -> Finding free connection: api.toodledo.com:80
34:49.154000 http -> Contacting host: api.toodledo.com:80
34:49.200000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
34:49.201000 http -> Request is:
POST /2/tasks/get.php HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.toodledo.com
Accept-charset: nil
Accept: */*
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)
Content-Type: application/x-www-form-urlencoded
Content-length: 212
f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
34:49.201000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<92>>)
34:49.264000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
34:49.264000 http -> data (point-max 1) (len 1380):
"HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 12:34:49 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1189
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328790882</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status>
<note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</!
priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
34:49.264000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<92>)
34:49.264000 http -> Saw end of headers... ( *http api.toodledo.com:80*<92>) @ 197
34:49.264000 http -> url-http-clean-headers, end-of-headers now at 189
34:49.264000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>)
34:49.264000 http -> Got a content-length 1189, being smart about document end.
34:49.264000 http -> Calling initial content-length for extra data at end of headers
34:49.264000 http -> url-http-content-length-after-change-function 189 1381 1192
34:49.265000 http -> End of document found!
34:49.265000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
34:49.265000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<92>)
34:49.265000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>)
34:49.266000 http -> Parsed HTTP headers: class=2 status=200
34:49.266000 http -> Finished parsing HTTP headers: t
34:49.266000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
34:49.266000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<92>)
----------------------------------------
...cj
^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 12:49 ` Christopher J. White
@ 2012-02-09 13:07 ` Andreas Schwab
0 siblings, 0 replies; 8+ messages in thread
From: Andreas Schwab @ 2012-02-09 13:07 UTC (permalink / raw)
To: Christopher J. White; +Cc: 10768
"Christopher J. White" <chris@grierwhite.com> writes:
> I considered that and looked hard for it to be the server. In the data
> output the double quote chars add an additional slash "\" that is not
> present in the actual data.
You are right, I missed that.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White
2012-02-09 9:15 ` Andreas Schwab
@ 2012-02-09 15:43 ` Andreas Schwab
2012-02-09 18:46 ` Christopher J. White
2012-02-09 20:14 ` Christopher J. White
1 sibling, 2 replies; 8+ messages in thread
From: Andreas Schwab @ 2012-02-09 15:43 UTC (permalink / raw)
To: Christopher J. White; +Cc: 10768
Does this help?
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index b43ed76..140824f 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -352,11 +352,14 @@ request.")
;; Parsing routines
(defun url-http-clean-headers ()
"Remove trailing \r from header lines.
-This allows us to use `mail-fetch-field', etc."
+This allows us to use `mail-fetch-field', etc.
+Return the number of characters removed."
(declare (special url-http-end-of-headers))
- (goto-char (point-min))
- (while (re-search-forward "\r$" url-http-end-of-headers t)
- (replace-match "")))
+ (let ((end (marker-position url-http-end-of-headers)))
+ (goto-char (point-min))
+ (while (re-search-forward "\r$" url-http-end-of-headers t)
+ (replace-match ""))
+ (- end url-http-end-of-headers)))
(defun url-http-handle-authentication (proxy)
(declare (special status success url-http-method url-http-data
@@ -1051,7 +1054,7 @@ the end of the document."
(setq url-http-end-of-headers (set-marker (make-marker)
(point))
end-of-headers t)
- (url-http-clean-headers)))
+ (setq nd (- nd (url-http-clean-headers))))
(if (not end-of-headers)
;; Haven't seen the end of the headers yet, need to wait
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply related [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 15:43 ` Andreas Schwab
@ 2012-02-09 18:46 ` Christopher J. White
2012-02-09 20:14 ` Christopher J. White
1 sibling, 0 replies; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 18:46 UTC (permalink / raw)
To: Andreas Schwab; +Cc: 10768
Quick work! First past looks good, except you're missing one additional
paren after the last diff line:
- (url-http-clean-headers)))
+ (setq nd (- nd (url-http-clean-headers)))))
Let me run this through a number of cases and see how well it holds up
for a few days.
Thanks
...cj
On Thu, 09 Feb 2012 16:43:27 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:
> Does this help?
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index b43ed76..140824f 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -352,11 +352,14 @@ request.")
> ;; Parsing routines
> (defun url-http-clean-headers ()
> "Remove trailing \r from header lines.
> -This allows us to use `mail-fetch-field', etc."
> +This allows us to use `mail-fetch-field', etc.
> +Return the number of characters removed."
> (declare (special url-http-end-of-headers))
> - (goto-char (point-min))
> - (while (re-search-forward "\r$" url-http-end-of-headers t)
> - (replace-match "")))
> + (let ((end (marker-position url-http-end-of-headers)))
> + (goto-char (point-min))
> + (while (re-search-forward "\r$" url-http-end-of-headers t)
> + (replace-match ""))
> + (- end url-http-end-of-headers)))
>
> (defun url-http-handle-authentication (proxy)
> (declare (special status success url-http-method url-http-data
> @@ -1051,7 +1054,7 @@ the end of the document."
> (setq url-http-end-of-headers (set-marker (make-marker)
> (point))
> end-of-headers t)
> - (url-http-clean-headers)))
> + (setq nd (- nd (url-http-clean-headers))))
>
> (if (not end-of-headers)
> ;; Haven't seen the end of the headers yet, need to wait
>
> Andreas.
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 15:43 ` Andreas Schwab
2012-02-09 18:46 ` Christopher J. White
@ 2012-02-09 20:14 ` Christopher J. White
2012-02-10 11:47 ` Andreas Schwab
1 sibling, 1 reply; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 20:14 UTC (permalink / raw)
To: Andreas Schwab; +Cc: 10768
Hi Andreas,
I have created a small set of files with lengths varying from
1200 to 1225, and from 2658 to 2662 and put them up on my personal web
server. (e.g. http://www.grierwhite.com/url-http-1220.txt)
I chose lengths that just barely stretch from 1 to 2 payload packets,
and from 2 to 3 packets.
1200-1207 -- 1 packets
1208-1225 -- 2 packets, 1-18 bytes in the 2nd
2658-2659 -- 2 packets
2660-2662 -- 3 packets, 1-3 bytes in the 3rd
Prior to the change, the results are as follows:
1200-1207 -- pass
1208-1217 -- fail
1217-1225 -- pass
2658-2662 -- pass
This indicates that the failure occurs when there are up to 10 bytes
in the 2nd packet. This correlates with the fact that the header is 10
lines, thus will replace 10 \r characters and shorten the buffer by
that amount. Consistent with the results I found with the original
server toodledo.com that only has 8 header lines and only has the issue
up to 8 bytes in the 2nd packet.
It is interesting to note that when there are 3 packets and just a few
bytes in the last packet, the failure does *not* occur. I think this
may be explained by the fact that
url-http-wait-for-headers-change-function may be called a second time,
thus all position references are recomputed. Andreas, can you verify
this logic? If this is correct, the problem only occurs for responses
that just border on 2 packets.
After the change, all lengths pass.
Here is the lisp code that tests the lengths, feel free to run this as
you should be able to reproduce this from any machine that can access
the internet. I don't have a trunk version of emacs (24), but looking
at the trunk copy of url-http.el, it appears to suffer from the same
problem.
(defun test-url-http-patch (minlen maxlen)
(let ((passed 0) (failed 0))
(do ((len minlen (+ 1 len))) ((> len maxlen) nil)
(set-buffer
(url-retrieve-synchronously
(format "http://www.grierwhite.com/url-http-%d.txt" len)))
(goto-char (point-min))
(re-search-forward "^\n")
(let ((recvlen (- (point-max) (point))))
(if (= len recvlen)
(setq passed (1+ passed))
(setq failed (1+ failed))
(message "FAILED orig-len: %d, received len: %d"
len recvlen))))
(message "Passed: %d/%d" passed (+ passed failed))))
;; Run without the patch
(test-url-http-patch 1200 1225)
"Passed: 16/26"
(test-url-http-patch 2658 2662)
"Passed: 5/5"
;; Run with the patch
(test-url-http-patch 1200 1225)
"Passed: 26/26"
(test-url-http-patch 2658 2662)
"Passed: 5/5"
...cj
On Thu, 09 Feb 2012 16:43:27 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:
> Does this help?
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index b43ed76..140824f 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -352,11 +352,14 @@ request.")
> ;; Parsing routines
> (defun url-http-clean-headers ()
> "Remove trailing \r from header lines.
> -This allows us to use `mail-fetch-field', etc."
> +This allows us to use `mail-fetch-field', etc.
> +Return the number of characters removed."
> (declare (special url-http-end-of-headers))
> - (goto-char (point-min))
> - (while (re-search-forward "\r$" url-http-end-of-headers t)
> - (replace-match "")))
> + (let ((end (marker-position url-http-end-of-headers)))
> + (goto-char (point-min))
> + (while (re-search-forward "\r$" url-http-end-of-headers t)
> + (replace-match ""))
> + (- end url-http-end-of-headers)))
>
> (defun url-http-handle-authentication (proxy)
> (declare (special status success url-http-method url-http-data
> @@ -1051,7 +1054,7 @@ the end of the document."
> (setq url-http-end-of-headers (set-marker (make-marker)
> (point))
> end-of-headers t)
> - (url-http-clean-headers)))
> + (setq nd (- nd (url-http-clean-headers))))
>
> (if (not end-of-headers)
> ;; Haven't seen the end of the headers yet, need to wait
>
> Andreas.
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
2012-02-09 20:14 ` Christopher J. White
@ 2012-02-10 11:47 ` Andreas Schwab
0 siblings, 0 replies; 8+ messages in thread
From: Andreas Schwab @ 2012-02-10 11:47 UTC (permalink / raw)
To: Christopher J. White; +Cc: 10768-done
Thanks for testing, installed in trunk.
Andreas.
--
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-02-10 11:47 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-02-09 4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White
2012-02-09 9:15 ` Andreas Schwab
2012-02-09 12:49 ` Christopher J. White
2012-02-09 13:07 ` Andreas Schwab
2012-02-09 15:43 ` Andreas Schwab
2012-02-09 18:46 ` Christopher J. White
2012-02-09 20:14 ` Christopher J. White
2012-02-10 11:47 ` Andreas Schwab
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).