* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used
@ 2012-02-09 4:53 Christopher J. White
2012-02-09 9:15 ` Andreas Schwab
2012-02-09 15:43 ` Andreas Schwab
0 siblings, 2 replies; 8+ messages in thread
From: Christopher J. White @ 2012-02-09 4:53 UTC (permalink / raw)
To: 10768
I’m using url-http to retrieve an XML response. In noticed somewhat by
chance that certain page sizes were cutting off the last few characters
of the response, which in turn caused the XML to be poorly formed.
I've noticed the issue occurs when the payload response has to be sent
in two TCP packets and the last packet has 8 or fewer bytes. I observe
this by looking at the HTTP stream in Wireshark.
I spent time tracing the response by adding some debug using
url-http-debug. The first chunk of data comes from the first TCP packet
and includes the HTTP header and all the XML data minus the last 8 bytes.
url-http-wait-for-headers-change-function is called to process this
chunk of data. After processing this chunk, the connection is closed
and the buffer of XML data returned minux the 8 bytes.
Here's the http trace, along with my added debug messages (inlcuding
dumping of the full data):
----------------------------------------
22:06.781000 http -> Contacting host: api.toodledo.com:80
22:06.827000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.828000 http -> Request is:
POST /2/tasks/get.php HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.toodledo.com
Accept-charset: nil
Accept: */*
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)
Content-Type: application/x-www-form-urlencoded
Content-length: 212
f=xml&key=9f2a92de083e8d63c2bd14771661139f&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
22:06.828000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<80>>)
22:06.889000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
22:06.889000 http -> data (point-max 1) (len 1380):
"HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 04:22:06 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1189
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></"
22:06.889000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<80>)
22:06.890000 http -> Saw end of headers... ( *http api.toodledo.com:80*<80>) @ 197
22:06.890000 http -> url-http-clean-headers, end-of-headers now at 189
22:06.890000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<80>)
22:06.890000 http -> Got a content-length 1189, being smart about document end.
22:06.890000 http -> Calling initial content-length for extra data at end of headers
22:06.890000 http -> url-http-content-length-after-change-function 189 1381 1192
22:06.891000 http -> End of document found!
22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.891000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<80>)
22:06.891000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<80>)
22:06.891000 http -> Parsed HTTP headers: class=2 status=200
22:06.891000 http -> Finished parsing HTTP headers: t
22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
22:06.891000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<80>)
----------------------------------------
Notice that the XML in the data finishes with "...</note></task></". It
should finish with: "...</note></task></tasks>".
A key debug message I added is "End of document found!", and this is
placed in the function url-http-content-length-after-change-function
right after the comment "Found the end of the document! Wheee!":
(defun url-http-content-length-after-change-function (st nd length)
<....>
(if (> (- nd url-http-end-of-headers) url-http-content-length)
(progn
;; Found the end of the document! Wheee!
(url-http-debug "End of document found!")
(url-display-percentage nil nil)
(url-lazy-message "Reading... done.")
(if (url-http-parse-headers)
(url-http-activate-callback)))))
If I slightly increase or decrease the amount of data returned
by the XML (by adding or deleting characters from title), I can get it
to work properly or not.
For example, by adding 5 chars, I get:
----------------------------------------
39:20.144000 http -> Finding free connection: api.toodledo.com:80
39:20.144000 http -> Contacting host: api.toodledo.com:80
39:20.187000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.188000 http -> Request is:
POST /2/tasks/get.php HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.toodledo.com
Accept-charset: nil
Accept: */*
User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)
Content-Type: application/x-www-form-urlencoded
Content-length: 212
f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0
39:20.188000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<86>>)
39:20.242000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>'
39:20.243000 http -> data (point-max 1) (len 1380):
"HTTP/1.1 200 OK
Date: Thu, 09 Feb 2012 04:39:20 GMT
Server: Apache/2.2.3 (Red Hat)
Content-Length: 1194
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/xml
<?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar12345</title><modified>1328762353</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></ta"
39:20.243000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<86>)
39:20.243000 http -> Saw end of headers... ( *http api.toodledo.com:80*<86>) @ 197
39:20.243000 http -> url-http-clean-headers, end-of-headers now at 189
39:20.243000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<86>)
39:20.243000 http -> Got a content-length 1194, being smart about document end.
39:20.243000 http -> Calling initial content-length for extra data at end of headers
39:20.243000 http -> url-http-content-length-after-change-function 189 1381 1192
39:20.244000 http -> Spinning waiting for headers...
39:20.244000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<86>>)
39:20.247000 http -> Calling after change function `url-http-content-length-after-change-function' for `#<process api.toodledo.com<1>>'
39:20.247000 http -> data (point-max 1373) (len 11):
"sk></tasks>"
39:20.247000 http -> url-http-content-length-after-change-function 1373 1384 11
39:20.247000 http -> End of document found!
39:20.247000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.247000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<86>)
39:20.247000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<86>)
39:20.247000 http -> Parsed HTTP headers: class=2 status=200
39:20.248000 http -> Finished parsing HTTP headers: t
39:20.248000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>>
39:20.248000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<86>)
----------------------------------------
In this case, url-http waits for the remaining data from the second TCP
packet, which comes in slightly later and is pieced together and forms a
good XML document.
If I reduce the size by an appropriate number of bytes so that the XML
data fits entirely in the first packet it also works.
I think the problem relates to url-http-clean-headers. I have 8-lines
of headers, and this function removes the trailing \r from each of those
lines. I think this is not accounted for when using the content-length
to determine if all the data has been received. After dropping these 8
trailing \r charaters, the buffer positions all change.
I'm not quite able to figure out how the content-length is used to
detect the end of processing, so I thought I'd submit this bug and see
if any other folks can point me in the right direction.
...cj
In GNU Emacs 23.3.1 (i386-mingw-nt6.1.7601)
of 2011-03-10 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 6.1.7601
configured using `configure --with-gcc (4.5) --no-opt --cflags -Ic:/imagesupport/include'
Important settings:
value of $LC_ALL: nil
value of $LC_COLLATE: nil
value of $LC_CTYPE: nil
value of $LC_MESSAGES: nil
value of $LC_MONETARY: nil
value of $LC_NUMERIC: nil
value of $LC_TIME: nil
value of $LANG: C.UTF-8
value of $XMODIFIERS: nil
locale-coding-system: cp1252
default enable-multibyte-characters: t
Major mode: Fundamental
Minor modes in effect:
show-paren-mode: t
display-time-mode: t
tooltip-mode: t
mouse-wheel-mode: t
global-font-lock-mode: t
auto-encryption-mode: t
auto-compression-mode: t
column-number-mode: t
line-number-mode: t
transient-mark-mode: t
Recent input:
- - - - - - - - - - - - - - - - <return> <return> <switch-frame>
<help-echo> <select-window> <backspace> <escape> C-x
<select-window> <help-echo> C-/ C-/ C-/ C-/ SPC C-/
C-/ C-a <tab> <tab> C-x [ C-x ] C-x [ C-/ C-/ C-/ C-x
[ C-SPC C-x ] C-w C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-a
<tab> <S-tab> <tab> C-o s <help-echo> <switch-frame>
C-SPC C-x ] <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <down> C-w <up> <down> <switch-frame>
<switch-frame> <down> <down> C-x 2 <prior> <prior>
<down> <down> <down-mouse-1> <mouse-movement> <mouse-1>
<C-left> C-s C-w C-w C-w C-w C-w C-w C-w <return> C-x
o C-s C-s C-a <help-echo> C-a C-l C-n C-n C-n C-n C-n
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n <down>
<down> <down> <down> C-x o C-p C-n C-n C-n C-n C-n
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-l
C-n C-n C-l C-n C-n C-n C-n C-n C-l C-n C-n C-n C-n
C-l C-x o C-n C-n C-n <select-window> <select-window>
<select-window> <help-echo> <prior> <prior> <next>
<switch-frame> <down-mouse-1> <mouse-movement> <mouse-1>
C-o s <select-window> <help-echo> <switch-frame> <down-mouse-1>
<mouse-movement> <mouse-1> C-x ] <prior> <switch-frame>
<down-mouse-1> <mouse-movement> <mouse-1> <down-mouse-1>
<mouse-1> SPC <escape> C-x <select-window> <help-echo>
<select-window> C-x o C-o s <switch-frame> <down-mouse-1>
<mouse-movement> <mouse-1> C-x ] C-r t a s k s C-r
C-r C-r C-s C-a <next> <switch-frame> <switch-frame>
<switch-frame> <switch-frame> <switch-frame> <switch-frame>
<switch-frame> <switch-frame> <escape> x r e p o <tab>
r t SPC e m SPC SPC <return>
Recent messages:
Reading [application/xml
]... 1k of 1k (100%)
Contacting host: api.toodledo.com:80
Reading [application/xml
]... 718 bytes of 717 bytes (100%)
Sync complete, 794 changes: recv 3 mod 791 del
Mark set
Mark saved where search started
Auto-saving...done
Making completion list...
Load-path shadows:
~cwhite/lib/lisp/org-toodledo/w3mexcerpt hides ~cwhite/lib/lisp/w3mexcerpt
~cwhite/lib/lisp/org-toodledo/http-post-simple hides ~cwhite/lib/lisp/http-post-simple
~/lib/lisp/org-mode/lisp/org hides c:/Users/cwhite/emacs-23.3/lisp/org/org
~/lib/lisp/org-mode/lisp/org-xoxo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-xoxo
~/lib/lisp/org-mode/lisp/org-wl hides c:/Users/cwhite/emacs-23.3/lisp/org/org-wl
~/lib/lisp/org-mode/lisp/org-w3m hides c:/Users/cwhite/emacs-23.3/lisp/org/org-w3m
~/lib/lisp/org-mode/lisp/org-vm hides c:/Users/cwhite/emacs-23.3/lisp/org/org-vm
~/lib/lisp/org-mode/lisp/org-timer hides c:/Users/cwhite/emacs-23.3/lisp/org/org-timer
~/lib/lisp/org-mode/lisp/org-table hides c:/Users/cwhite/emacs-23.3/lisp/org/org-table
~/lib/lisp/org-mode/lisp/org-src hides c:/Users/cwhite/emacs-23.3/lisp/org/org-src
~/lib/lisp/org-mode/lisp/org-rmail hides c:/Users/cwhite/emacs-23.3/lisp/org/org-rmail
~/lib/lisp/org-mode/lisp/org-remember hides c:/Users/cwhite/emacs-23.3/lisp/org/org-remember
~/lib/lisp/org-mode/lisp/org-publish hides c:/Users/cwhite/emacs-23.3/lisp/org/org-publish
~/lib/lisp/org-mode/lisp/org-protocol hides c:/Users/cwhite/emacs-23.3/lisp/org/org-protocol
~/lib/lisp/org-mode/lisp/org-plot hides c:/Users/cwhite/emacs-23.3/lisp/org/org-plot
~/lib/lisp/org-mode/lisp/org-mouse hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mouse
~/lib/lisp/org-mode/lisp/org-mobile hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mobile
~/lib/lisp/org-mode/lisp/org-mhe hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mhe
~/lib/lisp/org-mode/lisp/org-mew hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mew
~/lib/lisp/org-mode/lisp/org-macs hides c:/Users/cwhite/emacs-23.3/lisp/org/org-macs
~/lib/lisp/org-mode/lisp/org-mac-message hides c:/Users/cwhite/emacs-23.3/lisp/org/org-mac-message
~/lib/lisp/org-mode/lisp/org-list hides c:/Users/cwhite/emacs-23.3/lisp/org/org-list
~/lib/lisp/org-mode/lisp/org-latex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-latex
~/lib/lisp/org-mode/lisp/org-jsinfo hides c:/Users/cwhite/emacs-23.3/lisp/org/org-jsinfo
~/lib/lisp/org-mode/lisp/org-irc hides c:/Users/cwhite/emacs-23.3/lisp/org/org-irc
~/lib/lisp/org-mode/lisp/org-install hides c:/Users/cwhite/emacs-23.3/lisp/org/org-install
~/lib/lisp/org-mode/lisp/org-inlinetask hides c:/Users/cwhite/emacs-23.3/lisp/org/org-inlinetask
~/lib/lisp/org-mode/lisp/org-info hides c:/Users/cwhite/emacs-23.3/lisp/org/org-info
~/lib/lisp/org-mode/lisp/org-indent hides c:/Users/cwhite/emacs-23.3/lisp/org/org-indent
~/lib/lisp/org-mode/lisp/org-id hides c:/Users/cwhite/emacs-23.3/lisp/org/org-id
~/lib/lisp/org-mode/lisp/org-icalendar hides c:/Users/cwhite/emacs-23.3/lisp/org/org-icalendar
~/lib/lisp/org-mode/lisp/org-html hides c:/Users/cwhite/emacs-23.3/lisp/org/org-html
~/lib/lisp/org-mode/lisp/org-habit hides c:/Users/cwhite/emacs-23.3/lisp/org/org-habit
~/lib/lisp/org-mode/lisp/org-gnus hides c:/Users/cwhite/emacs-23.3/lisp/org/org-gnus
~/lib/lisp/org-mode/lisp/org-freemind hides c:/Users/cwhite/emacs-23.3/lisp/org/org-freemind
~/lib/lisp/org-mode/lisp/org-footnote hides c:/Users/cwhite/emacs-23.3/lisp/org/org-footnote
~/lib/lisp/org-mode/lisp/org-feed hides c:/Users/cwhite/emacs-23.3/lisp/org/org-feed
~/lib/lisp/org-mode/lisp/org-faces hides c:/Users/cwhite/emacs-23.3/lisp/org/org-faces
~/lib/lisp/org-mode/lisp/org-exp hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp
~/lib/lisp/org-mode/lisp/org-exp-blocks hides c:/Users/cwhite/emacs-23.3/lisp/org/org-exp-blocks
~/lib/lisp/org-mode/lisp/org-docbook hides c:/Users/cwhite/emacs-23.3/lisp/org/org-docbook
~/lib/lisp/org-mode/lisp/org-datetree hides c:/Users/cwhite/emacs-23.3/lisp/org/org-datetree
~/lib/lisp/org-mode/lisp/org-crypt hides c:/Users/cwhite/emacs-23.3/lisp/org/org-crypt
~/lib/lisp/org-mode/lisp/org-compat hides c:/Users/cwhite/emacs-23.3/lisp/org/org-compat
~/lib/lisp/org-mode/lisp/org-colview hides c:/Users/cwhite/emacs-23.3/lisp/org/org-colview
~/lib/lisp/org-mode/lisp/org-clock hides c:/Users/cwhite/emacs-23.3/lisp/org/org-clock
~/lib/lisp/org-mode/lisp/org-bibtex hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bibtex
~/lib/lisp/org-mode/lisp/org-bbdb hides c:/Users/cwhite/emacs-23.3/lisp/org/org-bbdb
~/lib/lisp/org-mode/lisp/org-attach hides c:/Users/cwhite/emacs-23.3/lisp/org/org-attach
~/lib/lisp/org-mode/lisp/org-ascii hides c:/Users/cwhite/emacs-23.3/lisp/org/org-ascii
~/lib/lisp/org-mode/lisp/org-archive hides c:/Users/cwhite/emacs-23.3/lisp/org/org-archive
~/lib/lisp/org-mode/lisp/org-agenda hides c:/Users/cwhite/emacs-23.3/lisp/org/org-agenda
Features:
(shadow sort mail-extr message ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailabbrev nnheader
gnus-util netrc gmm-utils mailheader canlock sha1 hex-util hashcash
emacsbug sgml-mode cl-specs edebug find-func vc-git nxml-uchnm rng-xsd
xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok browse-url
cperl-mode url-cache org-colview diary-lib diary-loaddefs cal-iso
org-table bookmark pp image-file mule-util grep sh-script executable
tabify org-ascii help-mode view org-wl org-w3m org-vm org-rmail org-mhe
org-mew org-irc org-jsinfo org-infojs org-html format-spec org-exp
ob-exp org-exp-blocks org-agenda org-info org-gnus org-docview
org-bibtex bibtex org-bbdb multi-isearch newcomment org-toodledo
http-post-simple url-http tls url-auth mail-parse rfc2231 rfc2047
rfc2045 qp ietf-drums url-gw url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-util url-parse url-vars mm-util
mail-prsvr mailcap json xml w3mexcerpt remember org-remember
org-datetree org ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table
org-footnote org-src ob-comint ob-keys ob ob-eval org-pcomplete
pcomplete org-list org-faces org-compat org-entities org-macs time-date
noutline outline easy-mmode cal-menu calendar cal-loaddefs cygwin-mount
ange-ftp uniquify filecache server byte-opt warnings advice help-fns
advice-preload paren time etags python-mode info-look css-mode scr rsync
twiki edmacro kmacro info doku pib php-mode speedbar sb-image ezimage
dframe assoc highline pop3 mail-utils cl cl-19 cc-mode cc-fonts cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs regexp-opt cus-edit
easymenu cus-start cus-load wid-edit exec compile comint ring bytecomp
byte-compile tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32
disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty emacs)
^ permalink raw reply [flat|nested] 8+ messages in thread* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White @ 2012-02-09 9:15 ` Andreas Schwab 2012-02-09 12:49 ` Christopher J. White 2012-02-09 15:43 ` Andreas Schwab 1 sibling, 1 reply; 8+ messages in thread From: Andreas Schwab @ 2012-02-09 9:15 UTC (permalink / raw) To: Christopher J. White; +Cc: 10768 "Christopher J. White" <chris@grierwhite.com> writes: > "HTTP/1.1 200 OK > Date: Thu, 09 Feb 2012 04:22:06 GMT > Server: Apache/2.2.3 (Red Hat) > Content-Length: 1189 ^^^^ > Keep-Alive: timeout=2, max=100 > Connection: Keep-Alive > Content-Type: application/xml > > <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></" 1189 bytes of data follow. That looks like a bug in the server. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 9:15 ` Andreas Schwab @ 2012-02-09 12:49 ` Christopher J. White 2012-02-09 13:07 ` Andreas Schwab 0 siblings, 1 reply; 8+ messages in thread From: Christopher J. White @ 2012-02-09 12:49 UTC (permalink / raw) To: Andreas Schwab; +Cc: 10768 On Thu, 09 Feb 2012 10:15:47 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: > "Christopher J. White" <chris@grierwhite.com> writes: > > Content-Length: 1189 > ^^^^ > > <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' > > total='3'><task><id>9105869</id><title>Test > > 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test > > 2 > > -foobar</title><modified>1328761322</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></" > > 1189 bytes of data follow. > > That looks like a bug in the server. I considered that and looked hard for it to be the server. In the data output the double quote chars add an additional slash "\" that is not present in the actual data. The content-length and the actual number of raw bytes matches precisely when decoded from wireshark. I reran the test and collected the 3 sets of data, the recombined HTTM response as built by wireshark (TCP Follow Stream), the ascii dump of the wireshark packets (2 involved), and the URL-Debug. The key point is that the data in the second packet is never processed by url-http, only the bytes in the first packet. HTTP response as seen in Wireshark: ---------------------------------------- f=xml&key=444ed11d926131624b2ce2ed81f58ad4&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0 HTTP/1.1 200 OK Date: Thu, 09 Feb 2012 12:34:49 GMT Server: Apache/2.2.3 (Red Hat) Content-Length: 1189 Keep-Alive: timeout=2, max=100 Connection: Keep-Alive Content-Type: application/xml <?xml version="1.0" encoding="UTF-8"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328790882</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><not e></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</prio! rity><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></tasks> ---------------------------------------- There is no newline at the end, the last character is the ">" after tasks. The xml data is all a single line, there are no CR/LF characters any where in it, so if any show up in this message they should be removed if you're trying to count it out. Here's the wireshark decode for the 2 packets involved: ---------------------------------------- No. Time Source Destination Protocol Length Info 312 49.870066 72.4.112.215 11.1.1.89 TCP 1434 [TCP segment of a reassembled PDU] Frame 312: 1434 bytes on wire (11472 bits), 1434 bytes captured (11472 bits) Arrival Time: Feb 9, 2012 07:34:49.261829000 Eastern Standard Time Epoch Time: 1328790889.261829000 seconds [Time delta from previous captured frame: 0.008369000 seconds] [Time delta from previous displayed frame: 0.008369000 seconds] [Time since reference or first frame: 49.870066000 seconds] Frame Number: 312 Frame Length: 1434 bytes (11472 bits) Capture Length: 1434 bytes (11472 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:tcp] [Coloring Rule Name: HTTP] [Coloring Rule String: http || tcp.port == 80] Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea) Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 1420 Identification: 0xe548 (58696) Flags: 0x02 (Don't Fragment) 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 55 Protocol: TCP (6) Header checksum: 0x93ee [correct] [Good: True] [Bad: False] Source: 72.4.112.215 (72.4.112.215) Destination: 11.1.1.89 (11.1.1.89) Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1, Ack: 527, Len: 1380 Source port: http (80) Destination port: 60823 (60823) [Stream index: 10] Sequence number: 1 (relative sequence number) [Next sequence number: 1381 (relative sequence number)] Acknowledgement number: 527 (relative ack number) Header length: 20 bytes Flags: 0x10 (ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgement: Set .... .... 0... = Push: Not set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set Window size value: 54 [Calculated window size: 6912] [Window size scaling factor: 128] Checksum: 0x836f [validation disabled] [Good Checksum: False] [Bad Checksum: False] [SEQ/ACK analysis] [Bytes in flight: 1380] TCP segment data (1380 bytes) 0000 00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00 .!k^.....D....E. 0010 05 8c e5 48 40 00 37 06 93 ee 48 04 70 d7 0b 01 ...H@.7...H.p... 0020 01 59 00 50 ed 97 30 cd 5f f0 eb ef 69 24 50 10 .Y.P..0._...i$P. 0030 00 36 83 6f 00 00 48 54 54 50 2f 31 2e 31 20 32 .6.o..HTTP/1.1 2 0040 30 30 20 4f 4b 0d 0a 44 61 74 65 3a 20 54 68 75 00 OK..Date: Thu 0050 2c 20 30 39 20 46 65 62 20 32 30 31 32 20 31 32 , 09 Feb 2012 12 0060 3a 33 34 3a 34 39 20 47 4d 54 0d 0a 53 65 72 76 :34:49 GMT..Serv 0070 65 72 3a 20 41 70 61 63 68 65 2f 32 2e 32 2e 33 er: Apache/2.2.3 0080 20 28 52 65 64 20 48 61 74 29 0d 0a 43 6f 6e 74 (Red Hat)..Cont 0090 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 31 38 39 ent-Length: 1189 00a0 0d 0a 4b 65 65 70 2d 41 6c 69 76 65 3a 20 74 69 ..Keep-Alive: ti 00b0 6d 65 6f 75 74 3d 32 2c 20 6d 61 78 3d 31 30 30 meout=2, max=100 00c0 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 65 ..Connection: Ke 00d0 65 70 2d 41 6c 69 76 65 0d 0a 43 6f 6e 74 65 6e ep-Alive..Conten 00e0 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat 00f0 69 6f 6e 2f 78 6d 6c 0d 0a 0d 0a 3c 3f 78 6d 6c ion/xml....<?xml 0100 20 76 65 72 73 69 6f 6e 3d 22 31 2e 30 22 20 65 version="1.0" e 0110 6e 63 6f 64 69 6e 67 3d 22 55 54 46 2d 38 22 3f ncoding="UTF-8"? 0120 3e 3c 74 61 73 6b 73 20 6e 75 6d 3d 27 33 27 20 ><tasks num='3' 0130 74 6f 74 61 6c 3d 27 33 27 3e 3c 74 61 73 6b 3e total='3'><task> 0140 3c 69 64 3e 39 31 30 35 38 36 39 3c 2f 69 64 3e <id>9105869</id> 0150 3c 74 69 74 6c 65 3e 54 65 73 74 20 31 3c 2f 74 <title>Test 1</t 0160 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e 31 itle><modified>1 0170 33 32 38 34 38 32 39 33 38 3c 2f 6d 6f 64 69 66 328482938</modif 0180 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e 31 ied><completed>1 0190 33 32 37 39 32 34 38 30 30 3c 2f 63 6f 6d 70 6c 327924800</compl 01a0 65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f eted><folder>0</ 01b0 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e folder><context> 01c0 30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 0</context><goal 01d0 3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 >0</goal><priori 01e0 74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e 3c ty>0</priority>< 01f0 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e repeat></repeat> 0200 3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72 <repeatfrom>0</r 0210 65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61 epeatfrom><dueda 0220 74 65 3e 31 33 32 38 30 39 37 36 30 30 3c 2f 64 te>1328097600</d 0230 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74 uedate><startdat 0240 65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c e>0</startdate>< 0250 64 75 65 74 69 6d 65 3e 31 33 32 38 31 33 37 32 duetime>13281372 0260 30 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 00</duetime><sta 0270 72 74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74 rttime>0</startt 0280 69 6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c ime><length>0</l 0290 65 6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c ength><status>2< 02a0 2f 73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f /status><note></ 02b0 6e 6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73 note></task><tas 02c0 6b 3e 3c 69 64 3e 39 31 30 38 33 30 39 3c 2f 69 k><id>9108309</i 02d0 64 3e 3c 74 69 74 6c 65 3e 54 65 73 74 20 32 20 d><title>Test 2 02e0 2d 66 6f 6f 62 61 72 3c 2f 74 69 74 6c 65 3e 3c -foobar</title>< 02f0 6d 6f 64 69 66 69 65 64 3e 31 33 32 38 37 39 30 modified>1328790 0300 38 38 32 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 882</modified><c 0310 6f 6d 70 6c 65 74 65 64 3e 30 3c 2f 63 6f 6d 70 ompleted>0</comp 0320 6c 65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c leted><folder>0< 0330 2f 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 /folder><context 0340 3e 30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 >0</context><goa 0350 6c 3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 l>0</goal><prior 0360 69 74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e ity>0</priority> 0370 3c 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 <repeat></repeat 0380 3e 3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f ><repeatfrom>0</ 0390 72 65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 repeatfrom><dued 03a0 61 74 65 3e 30 3c 2f 64 75 65 64 61 74 65 3e 3c ate>0</duedate>< 03b0 73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73 74 61 startdate>0</sta 03c0 72 74 64 61 74 65 3e 3c 64 75 65 74 69 6d 65 3e rtdate><duetime> 03d0 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 72 0</duetime><star 03e0 74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74 69 ttime>0</startti 03f0 6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65 me><length>0</le 0400 6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c 2f ngth><status>2</ 0410 73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e status><note></n 0420 6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73 6b ote></task><task 0430 3e 3c 69 64 3e 39 31 34 39 33 33 33 3c 2f 69 64 ><id>9149333</id 0440 3e 3c 74 69 74 6c 65 3e 6f 72 67 2d 74 6f 6f 64 ><title>org-tood 0450 6c 65 64 6f 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 ledo</title><mod 0460 69 66 69 65 64 3e 31 33 32 38 30 31 31 39 34 39 ified>1328011949 0470 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70 </modified><comp 0480 6c 65 74 65 64 3e 30 3c 2f 63 6f 6d 70 6c 65 74 leted>0</complet 0490 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f ed><folder>0</fo 04a0 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c lder><context>0< 04b0 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 /context><goal>0 04c0 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79 </goal><priority 04d0 3e 32 3c 2f 70 72 69 6f 72 69 74 79 3e 3c 72 65 >2</priority><re 04e0 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 peat></repeat><r 04f0 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 epeatfrom>0</rep 0500 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61 74 65 eatfrom><duedate 0510 3e 30 3c 2f 64 75 65 64 61 74 65 3e 3c 73 74 61 >0</duedate><sta 0520 72 74 64 61 74 65 3e 30 3c 2f 73 74 61 72 74 64 rtdate>0</startd 0530 61 74 65 3e 3c 64 75 65 74 69 6d 65 3e 30 3c 2f ate><duetime>0</ 0540 64 75 65 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 duetime><startti 0550 6d 65 3e 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e me>0</starttime> 0560 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 <length>0</lengt 0570 68 3e 3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 h><status>2</sta 0580 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 tus><note></note 0590 3e 3c 2f 74 61 73 6b 3e 3c 2f ></task></ No. Time Source Destination Protocol Length Info 313 49.874992 72.4.112.215 11.1.1.89 HTTP/XML 60 HTTP/1.1 200 OK Frame 313: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Arrival Time: Feb 9, 2012 07:34:49.266755000 Eastern Standard Time Epoch Time: 1328790889.266755000 seconds [Time delta from previous captured frame: 0.004926000 seconds] [Time delta from previous displayed frame: 0.004926000 seconds] [Time since reference or first frame: 49.874992000 seconds] Frame Number: 313 Frame Length: 60 bytes (480 bits) Capture Length: 60 bytes (480 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:tcp:http:xml] [Coloring Rule Name: HTTP] [Coloring Rule String: http || tcp.port == 80] Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea) Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 46 Identification: 0xe549 (58697) Flags: 0x02 (Don't Fragment) 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 55 Protocol: TCP (6) Header checksum: 0x994b [correct] [Good: True] [Bad: False] Source: 72.4.112.215 (72.4.112.215) Destination: 11.1.1.89 (11.1.1.89) Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1381, Ack: 527, Len: 6 Source port: http (80) Destination port: 60823 (60823) [Stream index: 10] Sequence number: 1381 (relative sequence number) [Next sequence number: 1387 (relative sequence number)] Acknowledgement number: 527 (relative ack number) Header length: 20 bytes Flags: 0x18 (PSH, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgement: Set .... .... 1... = Push: Set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set Window size value: 54 [Calculated window size: 6912] [Window size scaling factor: 128] Checksum: 0xb632 [validation disabled] [Good Checksum: False] [Bad Checksum: False] [SEQ/ACK analysis] [Bytes in flight: 1386] TCP segment data (6 bytes) [2 Reassembled TCP Segments (1386 bytes): #312(1380), #313(6)] [Frame: 312, payload: 0-1379 (1380 bytes)] [Frame: 313, payload: 1380-1385 (6 bytes)] [Segment count: 2] [Reassembled TCP length: 1386] Hypertext Transfer Protocol HTTP/1.1 200 OK\r\n [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n] [Message: HTTP/1.1 200 OK\r\n] [Severity level: Chat] [Group: Sequence] Request Version: HTTP/1.1 Status Code: 200 Response Phrase: OK Date: Thu, 09 Feb 2012 12:34:49 GMT\r\n Server: Apache/2.2.3 (Red Hat)\r\n Content-Length: 1189\r\n [Content length: 1189] Keep-Alive: timeout=2, max=100\r\n Connection: Keep-Alive\r\n Content-Type: application/xml\r\n \r\n eXtensible Markup Language <?xml version="1.0" encoding="UTF-8" ?> <tasks num='3' total='3'> <task> <id> 9105869 </id> <title> Test 1 </title> <modified> 1328482938 </modified> <completed> 1327924800 </completed> <folder> 0 </folder> <context> 0 </context> <goal> 0 </goal> <priority> 0 </priority> <repeat> </repeat> <repeatfrom> 0 </repeatfrom> <duedate> 1328097600 </duedate> <startdate> 0 </startdate> <duetime> 1328137200 </duetime> <starttime> 0 </starttime> <length> 0 </length> <status> 2 </status> <note> </note> </task> <task> <id> 9108309 </id> <title> Test 2 -foobar </title> <modified> 1328790882 </modified> <completed> 0 </completed> <folder> 0 </folder> <context> 0 </context> <goal> 0 </goal> <priority> 0 </priority> <repeat> </repeat> <repeatfrom> 0 </repeatfrom> <duedate> 0 </duedate> <startdate> 0 </startdate> <duetime> 0 </duetime> <starttime> 0 </starttime> <length> 0 </length> <status> 2 </status> <note> </note> </task> <task> <id> 9149333 </id> <title> org-toodledo </title> <modified> 1328011949 </modified> <completed> 0 </completed> <folder> 0 </folder> <context> 0 </context> <goal> 0 </goal> <priority> 2 </priority> <repeat> </repeat> <repeatfrom> 0 </repeatfrom> <duedate> 0 </duedate> <startdate> 0 </startdate> <duetime> 0 </duetime> <starttime> 0 </starttime> <length> 0 </length> <status> 2 </status> <note> </note> </task> </tasks> Frame (60 bytes): 0000 00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00 .!k^.....D....E. 0010 00 2e e5 49 40 00 37 06 99 4b 48 04 70 d7 0b 01 ...I@.7..KH.p... 0020 01 59 00 50 ed 97 30 cd 65 54 eb ef 69 24 50 18 .Y.P..0.eT..i$P. 0030 00 36 b6 32 00 00 74 61 73 6b 73 3e .6.2..tasks> Reassembled TCP (1386 bytes): 0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK. 0010 0a 44 61 74 65 3a 20 54 68 75 2c 20 30 39 20 46 .Date: Thu, 09 F 0020 65 62 20 32 30 31 32 20 31 32 3a 33 34 3a 34 39 eb 2012 12:34:49 0030 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 GMT..Server: Ap 0040 61 63 68 65 2f 32 2e 32 2e 33 20 28 52 65 64 20 ache/2.2.3 (Red 0050 48 61 74 29 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 Hat)..Content-Le 0060 6e 67 74 68 3a 20 31 31 38 39 0d 0a 4b 65 65 70 ngth: 1189..Keep 0070 2d 41 6c 69 76 65 3a 20 74 69 6d 65 6f 75 74 3d -Alive: timeout= 0080 32 2c 20 6d 61 78 3d 31 30 30 0d 0a 43 6f 6e 6e 2, max=100..Conn 0090 65 63 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 ection: Keep-Ali 00a0 76 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 ve..Content-Type 00b0 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d : application/xm 00c0 6c 0d 0a 0d 0a 3c 3f 78 6d 6c 20 76 65 72 73 69 l....<?xml versi 00d0 6f 6e 3d 22 31 2e 30 22 20 65 6e 63 6f 64 69 6e on="1.0" encodin 00e0 67 3d 22 55 54 46 2d 38 22 3f 3e 3c 74 61 73 6b g="UTF-8"?><task 00f0 73 20 6e 75 6d 3d 27 33 27 20 74 6f 74 61 6c 3d s num='3' total= 0100 27 33 27 3e 3c 74 61 73 6b 3e 3c 69 64 3e 39 31 '3'><task><id>91 0110 30 35 38 36 39 3c 2f 69 64 3e 3c 74 69 74 6c 65 05869</id><title 0120 3e 54 65 73 74 20 31 3c 2f 74 69 74 6c 65 3e 3c >Test 1</title>< 0130 6d 6f 64 69 66 69 65 64 3e 31 33 32 38 34 38 32 modified>1328482 0140 39 33 38 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 938</modified><c 0150 6f 6d 70 6c 65 74 65 64 3e 31 33 32 37 39 32 34 ompleted>1327924 0160 38 30 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 800</completed>< 0170 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72 folder>0</folder 0180 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e ><context>0</con 0190 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 6f text><goal>0</go 01a0 61 6c 3e 3c 70 72 69 6f 72 69 74 79 3e 30 3c 2f al><priority>0</ 01b0 70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 74 priority><repeat 01c0 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61 ></repeat><repea 01d0 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 66 tfrom>0</repeatf 01e0 72 6f 6d 3e 3c 64 75 65 64 61 74 65 3e 31 33 32 rom><duedate>132 01f0 38 30 39 37 36 30 30 3c 2f 64 75 65 64 61 74 65 8097600</duedate 0200 3e 3c 73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73 ><startdate>0</s 0210 74 61 72 74 64 61 74 65 3e 3c 64 75 65 74 69 6d tartdate><duetim 0220 65 3e 31 33 32 38 31 33 37 32 30 30 3c 2f 64 75 e>1328137200</du 0230 65 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 6d 65 etime><starttime 0240 3e 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c >0</starttime><l 0250 65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e ength>0</length> 0260 3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 <status>2</statu 0270 73 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c s><note></note>< 0280 2f 74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e /task><task><id> 0290 39 31 30 38 33 30 39 3c 2f 69 64 3e 3c 74 69 74 9108309</id><tit 02a0 6c 65 3e 54 65 73 74 20 32 20 2d 66 6f 6f 62 61 le>Test 2 -fooba 02b0 72 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 r</title><modifi 02c0 65 64 3e 31 33 32 38 37 39 30 38 38 32 3c 2f 6d ed>1328790882</m 02d0 6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 odified><complet 02e0 65 64 3e 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e ed>0</completed> 02f0 3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 <folder>0</folde 0300 72 3e 3c 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f r><context>0</co 0310 6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 ntext><goal>0</g 0320 6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79 3e 30 3c oal><priority>0< 0330 2f 70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 /priority><repea 0340 74 3e 3c 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 t></repeat><repe 0350 61 74 66 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 atfrom>0</repeat 0360 66 72 6f 6d 3e 3c 64 75 65 64 61 74 65 3e 30 3c from><duedate>0< 0370 2f 64 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 /duedate><startd 0380 61 74 65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 ate>0</startdate 0390 3e 3c 64 75 65 74 69 6d 65 3e 30 3c 2f 64 75 65 ><duetime>0</due 03a0 74 69 6d 65 3e 3c 73 74 61 72 74 74 69 6d 65 3e time><starttime> 03b0 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c 65 0</starttime><le 03c0 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e 3c ngth>0</length>< 03d0 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 73 status>2</status 03e0 3e 3c 6e 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c 2f ><note></note></ 03f0 74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e 39 task><task><id>9 0400 31 34 39 33 33 33 3c 2f 69 64 3e 3c 74 69 74 6c 149333</id><titl 0410 65 3e 6f 72 67 2d 74 6f 6f 64 6c 65 64 6f 3c 2f e>org-toodledo</ 0420 74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e title><modified> 0430 31 33 32 38 30 31 31 39 34 39 3c 2f 6d 6f 64 69 1328011949</modi 0440 66 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e fied><completed> 0450 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 66 6f 0</completed><fo 0460 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72 3e 3c lder>0</folder>< 0470 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e 74 65 context>0</conte 0480 78 74 3e 3c 67 6f 61 6c 3e 30 3c 2f 67 6f 61 6c xt><goal>0</goal 0490 3e 3c 70 72 69 6f 72 69 74 79 3e 32 3c 2f 70 72 ><priority>2</pr 04a0 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 74 3e 3c iority><repeat>< 04b0 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61 74 66 /repeat><repeatf 04c0 72 6f 6d 3e 30 3c 2f 72 65 70 65 61 74 66 72 6f rom>0</repeatfro 04d0 6d 3e 3c 64 75 65 64 61 74 65 3e 30 3c 2f 64 75 m><duedate>0</du 04e0 65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74 65 edate><startdate 04f0 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c 64 >0</startdate><d 0500 75 65 74 69 6d 65 3e 30 3c 2f 64 75 65 74 69 6d uetime>0</duetim 0510 65 3e 3c 73 74 61 72 74 74 69 6d 65 3e 30 3c 2f e><starttime>0</ 0520 73 74 61 72 74 74 69 6d 65 3e 3c 6c 65 6e 67 74 starttime><lengt 0530 68 3e 30 3c 2f 6c 65 6e 67 74 68 3e 3c 73 74 61 h>0</length><sta 0540 74 75 73 3e 32 3c 2f 73 74 61 74 75 73 3e 3c 6e tus>2</status><n 0550 6f 74 65 3e 3c 2f 6e 6f 74 65 3e 3c 2f 74 61 73 ote></note></tas 0560 6b 3e 3c 2f 74 61 73 6b 73 3e k></tasks> ---------------------------------------- And the URL-DEBUG: ---------------------------------------- 34:49.154000 http -> Finding free connection: api.toodledo.com:80 34:49.154000 http -> Contacting host: api.toodledo.com:80 34:49.200000 http -> Marking connection as busy: api.toodledo.com:80 #<process api.toodledo.com<1>> 34:49.201000 http -> Request is: POST /2/tasks/get.php HTTP/1.1 MIME-Version: 1.0 Connection: keep-alive Extension: Security/Digest Security/SSL Host: api.toodledo.com Accept-charset: nil Accept: */* User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit) Content-Type: application/x-www-form-urlencoded Content-length: 212 f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0 34:49.201000 retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer *http api.toodledo.com:80*<92>>) 34:49.264000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.toodledo.com<1>>' 34:49.264000 http -> data (point-max 1) (len 1380): "HTTP/1.1 200 OK Date: Thu, 09 Feb 2012 12:34:49 GMT Server: Apache/2.2.3 (Red Hat) Content-Length: 1189 Keep-Alive: timeout=2, max=100 Connection: Keep-Alive Content-Type: application/xml <?xml version=\"1.0\" encoding=\"UTF-8\"?><tasks num='3' total='3'><task><id>9105869</id><title>Test 1</title><modified>1328482938</modified><completed>1327924800</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>1328097600</duedate><startdate>0</startdate><duetime>1328137200</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task><task><id>9108309</id><title>Test 2 -foobar</title><modified>1328790882</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>0</priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status> <note></note></task><task><id>9149333</id><title>org-toodledo</title><modified>1328011949</modified><completed>0</completed><folder>0</folder><context>0</context><goal>0</goal><priority>2</! priority><repeat></repeat><repeatfrom>0</repeatfrom><duedate>0</duedate><startdate>0</startdate><duetime>0</duetime><starttime>0</starttime><length>0</length><status>2</status><note></note></task></" 34:49.264000 http -> url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<92>) 34:49.264000 http -> Saw end of headers... ( *http api.toodledo.com:80*<92>) @ 197 34:49.264000 http -> url-http-clean-headers, end-of-headers now at 189 34:49.264000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>) 34:49.264000 http -> Got a content-length 1189, being smart about document end. 34:49.264000 http -> Calling initial content-length for extra data at end of headers 34:49.264000 http -> url-http-content-length-after-change-function 189 1381 1192 34:49.265000 http -> End of document found! 34:49.265000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>> 34:49.265000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<92>) 34:49.265000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>) 34:49.266000 http -> Parsed HTTP headers: class=2 status=200 34:49.266000 http -> Finished parsing HTTP headers: t 34:49.266000 http -> Marking connection as free: api.toodledo.com:80 #<process api.toodledo.com<1>> 34:49.266000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<92>) ---------------------------------------- ...cj ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 12:49 ` Christopher J. White @ 2012-02-09 13:07 ` Andreas Schwab 0 siblings, 0 replies; 8+ messages in thread From: Andreas Schwab @ 2012-02-09 13:07 UTC (permalink / raw) To: Christopher J. White; +Cc: 10768 "Christopher J. White" <chris@grierwhite.com> writes: > I considered that and looked hard for it to be the server. In the data > output the double quote chars add an additional slash "\" that is not > present in the actual data. You are right, I missed that. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White 2012-02-09 9:15 ` Andreas Schwab @ 2012-02-09 15:43 ` Andreas Schwab 2012-02-09 18:46 ` Christopher J. White 2012-02-09 20:14 ` Christopher J. White 1 sibling, 2 replies; 8+ messages in thread From: Andreas Schwab @ 2012-02-09 15:43 UTC (permalink / raw) To: Christopher J. White; +Cc: 10768 Does this help? diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el index b43ed76..140824f 100644 --- a/lisp/url/url-http.el +++ b/lisp/url/url-http.el @@ -352,11 +352,14 @@ request.") ;; Parsing routines (defun url-http-clean-headers () "Remove trailing \r from header lines. -This allows us to use `mail-fetch-field', etc." +This allows us to use `mail-fetch-field', etc. +Return the number of characters removed." (declare (special url-http-end-of-headers)) - (goto-char (point-min)) - (while (re-search-forward "\r$" url-http-end-of-headers t) - (replace-match ""))) + (let ((end (marker-position url-http-end-of-headers))) + (goto-char (point-min)) + (while (re-search-forward "\r$" url-http-end-of-headers t) + (replace-match "")) + (- end url-http-end-of-headers))) (defun url-http-handle-authentication (proxy) (declare (special status success url-http-method url-http-data @@ -1051,7 +1054,7 @@ the end of the document." (setq url-http-end-of-headers (set-marker (make-marker) (point)) end-of-headers t) - (url-http-clean-headers))) + (setq nd (- nd (url-http-clean-headers)))) (if (not end-of-headers) ;; Haven't seen the end of the headers yet, need to wait Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply related [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 15:43 ` Andreas Schwab @ 2012-02-09 18:46 ` Christopher J. White 2012-02-09 20:14 ` Christopher J. White 1 sibling, 0 replies; 8+ messages in thread From: Christopher J. White @ 2012-02-09 18:46 UTC (permalink / raw) To: Andreas Schwab; +Cc: 10768 Quick work! First past looks good, except you're missing one additional paren after the last diff line: - (url-http-clean-headers))) + (setq nd (- nd (url-http-clean-headers))))) Let me run this through a number of cases and see how well it holds up for a few days. Thanks ...cj On Thu, 09 Feb 2012 16:43:27 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: > Does this help? > > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el > index b43ed76..140824f 100644 > --- a/lisp/url/url-http.el > +++ b/lisp/url/url-http.el > @@ -352,11 +352,14 @@ request.") > ;; Parsing routines > (defun url-http-clean-headers () > "Remove trailing \r from header lines. > -This allows us to use `mail-fetch-field', etc." > +This allows us to use `mail-fetch-field', etc. > +Return the number of characters removed." > (declare (special url-http-end-of-headers)) > - (goto-char (point-min)) > - (while (re-search-forward "\r$" url-http-end-of-headers t) > - (replace-match ""))) > + (let ((end (marker-position url-http-end-of-headers))) > + (goto-char (point-min)) > + (while (re-search-forward "\r$" url-http-end-of-headers t) > + (replace-match "")) > + (- end url-http-end-of-headers))) > > (defun url-http-handle-authentication (proxy) > (declare (special status success url-http-method url-http-data > @@ -1051,7 +1054,7 @@ the end of the document." > (setq url-http-end-of-headers (set-marker (make-marker) > (point)) > end-of-headers t) > - (url-http-clean-headers))) > + (setq nd (- nd (url-http-clean-headers)))) > > (if (not end-of-headers) > ;; Haven't seen the end of the headers yet, need to wait > > Andreas. > ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 15:43 ` Andreas Schwab 2012-02-09 18:46 ` Christopher J. White @ 2012-02-09 20:14 ` Christopher J. White 2012-02-10 11:47 ` Andreas Schwab 1 sibling, 1 reply; 8+ messages in thread From: Christopher J. White @ 2012-02-09 20:14 UTC (permalink / raw) To: Andreas Schwab; +Cc: 10768 Hi Andreas, I have created a small set of files with lengths varying from 1200 to 1225, and from 2658 to 2662 and put them up on my personal web server. (e.g. http://www.grierwhite.com/url-http-1220.txt) I chose lengths that just barely stretch from 1 to 2 payload packets, and from 2 to 3 packets. 1200-1207 -- 1 packets 1208-1225 -- 2 packets, 1-18 bytes in the 2nd 2658-2659 -- 2 packets 2660-2662 -- 3 packets, 1-3 bytes in the 3rd Prior to the change, the results are as follows: 1200-1207 -- pass 1208-1217 -- fail 1217-1225 -- pass 2658-2662 -- pass This indicates that the failure occurs when there are up to 10 bytes in the 2nd packet. This correlates with the fact that the header is 10 lines, thus will replace 10 \r characters and shorten the buffer by that amount. Consistent with the results I found with the original server toodledo.com that only has 8 header lines and only has the issue up to 8 bytes in the 2nd packet. It is interesting to note that when there are 3 packets and just a few bytes in the last packet, the failure does *not* occur. I think this may be explained by the fact that url-http-wait-for-headers-change-function may be called a second time, thus all position references are recomputed. Andreas, can you verify this logic? If this is correct, the problem only occurs for responses that just border on 2 packets. After the change, all lengths pass. Here is the lisp code that tests the lengths, feel free to run this as you should be able to reproduce this from any machine that can access the internet. I don't have a trunk version of emacs (24), but looking at the trunk copy of url-http.el, it appears to suffer from the same problem. (defun test-url-http-patch (minlen maxlen) (let ((passed 0) (failed 0)) (do ((len minlen (+ 1 len))) ((> len maxlen) nil) (set-buffer (url-retrieve-synchronously (format "http://www.grierwhite.com/url-http-%d.txt" len))) (goto-char (point-min)) (re-search-forward "^\n") (let ((recvlen (- (point-max) (point)))) (if (= len recvlen) (setq passed (1+ passed)) (setq failed (1+ failed)) (message "FAILED orig-len: %d, received len: %d" len recvlen)))) (message "Passed: %d/%d" passed (+ passed failed)))) ;; Run without the patch (test-url-http-patch 1200 1225) "Passed: 16/26" (test-url-http-patch 2658 2662) "Passed: 5/5" ;; Run with the patch (test-url-http-patch 1200 1225) "Passed: 26/26" (test-url-http-patch 2658 2662) "Passed: 5/5" ...cj On Thu, 09 Feb 2012 16:43:27 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: > Does this help? > > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el > index b43ed76..140824f 100644 > --- a/lisp/url/url-http.el > +++ b/lisp/url/url-http.el > @@ -352,11 +352,14 @@ request.") > ;; Parsing routines > (defun url-http-clean-headers () > "Remove trailing \r from header lines. > -This allows us to use `mail-fetch-field', etc." > +This allows us to use `mail-fetch-field', etc. > +Return the number of characters removed." > (declare (special url-http-end-of-headers)) > - (goto-char (point-min)) > - (while (re-search-forward "\r$" url-http-end-of-headers t) > - (replace-match ""))) > + (let ((end (marker-position url-http-end-of-headers))) > + (goto-char (point-min)) > + (while (re-search-forward "\r$" url-http-end-of-headers t) > + (replace-match "")) > + (- end url-http-end-of-headers))) > > (defun url-http-handle-authentication (proxy) > (declare (special status success url-http-method url-http-data > @@ -1051,7 +1054,7 @@ the end of the document." > (setq url-http-end-of-headers (set-marker (make-marker) > (point)) > end-of-headers t) > - (url-http-clean-headers))) > + (setq nd (- nd (url-http-clean-headers)))) > > (if (not end-of-headers) > ;; Haven't seen the end of the headers yet, need to wait > > Andreas. > ^ permalink raw reply [flat|nested] 8+ messages in thread
* bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used 2012-02-09 20:14 ` Christopher J. White @ 2012-02-10 11:47 ` Andreas Schwab 0 siblings, 0 replies; 8+ messages in thread From: Andreas Schwab @ 2012-02-10 11:47 UTC (permalink / raw) To: Christopher J. White; +Cc: 10768-done Thanks for testing, installed in trunk. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-02-10 11:47 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-02-09 4:53 bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Christopher J. White 2012-02-09 9:15 ` Andreas Schwab 2012-02-09 12:49 ` Christopher J. White 2012-02-09 13:07 ` Andreas Schwab 2012-02-09 15:43 ` Andreas Schwab 2012-02-09 18:46 ` Christopher J. White 2012-02-09 20:14 ` Christopher J. White 2012-02-10 11:47 ` Andreas Schwab
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).