23.0.50; url-retrieve-synchronously and Delicious API problem

23.0.50; url-retrieve-synchronously and Delicious API problem

[Mark Plaksin]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the emacs-pretest-bug@gnu.org mailing list.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

Run 'emacs -Q' and then evaluate this in *scratch*:
(url-retrieve-synchronously "https://api.del.icio.us/v1/posts/update")

Two " *http..." buffers are created.  The first contains a page with an
HTTP status of 401 saying that authentication is required.  The second
(after I type my username and password) contains the text of the page I
was trying to reach.

url-retrieve-synchronously returns the first buffer instead of the
second.  If I run url-retrieve-synchronously again the authentication is
cached so there's no 401 to worry about and everything works.

I can make it break over and over again if I empty the auth cache as
follows and try again:
(setq url-http-real-basic-auth-storage nil)

You'll need a del.icio.us username and password to test.  I haven't
found another site at which I can reproduce the problem.  I've never
been able to understand how url-retrieve{,-synchronously} really works
so I probably won't be much help at figuring out what's going on.

Here's the debug output (from *URL-DEBUG*):

http -> Contacting host: api.del.icio.us:443
http -> Marking connection as busy: api.del.icio.us:443 #<process api.del.icio.us>
http -> Request is: 
GET /v1/posts/update HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.del.icio.us
Accept-charset: iso-8859-1;q=1, big5;q=0.5, iso-2022-jp;q=0.5, shift_jis;q=0.5, utf-8;q=0.5, iso-8859-15;q=0.5, iso-8859-2;q=0.5, iso-8859-3;q=0.5, iso-8859-4;q=0.5, iso-8859-5;q=0.5, iso-8859-7;q=0.5, iso-8859-8;q=0.5, iso-8859-9;q=0.5, gb2312;q=0.5, eu\x04c-jp;q=0.5, euc-kr;q=0.5, tis-620;q=0.5, iso-8859-14;q=0.5, windows-1252;q=0.5, windows-1251;q=0.5, koi8-r;q=0.5, koi8-u;q=0.5, viscii;q=0.5, hz-gb-2312;q=0.5, iso-2022-cn-ext;q=0.5, iso-2022-cn;q=0.5, iso-2022-jp-2;q=0.5, iso-2022-kr;q=0.5, utf-16;q=0.5\x04, utf-16be;q=0.5, utf-16le;q=0.5
Accept: */*
User-Agent: URL/Emacs (i686-pc-linux-gnu; TTY)


retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer  *http api.del.icio.us:443*>)
http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.del.icio.us>'
http -> url-http-wait-for-headers-change-function ( *http api.del.icio.us:443*)
http -> Saw end of headers... ( *http api.del.icio.us:443*)
http -> url-http-parse-response called in ( *http api.del.icio.us:443*)
http -> Saw chunked encoding.
http -> Calling initial chunked-encoding for extra data at end of headers
http -> Reading chunk 0 (210 2576 2366)
http -> Saw start of chunk 1 (length=3462, start=211
http -> Reading chunk 1 (210 2576 2366)
http -> Still need 1097 bytes to hit end of chunk
http -> Still spinning for next chunk...
http -> Spinning waiting for headers...
http -> Calling after change function `url-http-chunked-encoding-after-change-function' for `#<process api.del.icio.us>'
http -> Reading chunk 1 (2560 3720 1160)
http -> Got to the end of chunk #1!
http -> Saw start of chunk 2 (length=0, start=3673
http -> Saw end of stream chunk!
http -> Saw end of trailers...
http -> Marking connection as free: api.del.icio.us:443 #<process api.del.icio.us>
http -> url-http-parse-headers called in ( *http api.del.icio.us:443*)
http -> url-http-parse-response called in ( *http api.del.icio.us:443*)
http -> Parsed HTTP headers: class=4 status=401
http -> Handling normal authentication
http -> Contacting host: api.del.icio.us:443
http -> Marking connection as busy: api.del.icio.us:443 #<process api.del.icio.us>
http -> Request is: 
GET /v1/posts/update HTTP/1.1
MIME-Version: 1.0
Connection: keep-alive
Extension: Security/Digest Security/SSL
Host: api.del.icio.us
Accept-charset: iso-8859-1;q=1, big5;q=0.5, iso-2022-jp;q=0.5, shift_jis;q=0.5, utf-8;q=0.5, iso-8859-15;q=0.5, iso-8859-2;q=0.5, iso-8859-3;q=0.5, iso-8859-4;q=0.5, iso-8859-5;q=0.5, iso-8859-7;q=0.5, iso-8859-8;q=0.5, iso-8859-9;q=0.5, gb2312;q=0.5, eu\x04c-jp;q=0.5, euc-kr;q=0.5, tis-620;q=0.5, iso-8859-14;q=0.5, windows-1252;q=0.5, windows-1251;q=0.5, koi8-r;q=0.5, koi8-u;q=0.5, viscii;q=0.5, hz-gb-2312;q=0.5, iso-2022-cn-ext;q=0.5, iso-2022-cn;q=0.5, iso-2022-jp-2;q=0.5, iso-2022-kr;q=0.5, utf-16;q=0.5\x04, utf-16be;q=0.5, utf-16le;q=0.5
Accept: */*
User-Agent: URL/Emacs (i686-pc-linux-gnu; TTY)
Authorization: Basic --DELETED--


http -> Finished parsing HTTP headers: nil
retrieval -> Spinning in url-retrieve-synchronously: nil (#<buffer  *http api.del.icio.us:443*>)
http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process api.del.icio.us>'
http -> url-http-wait-for-headers-change-function ( *http api.del.icio.us:443*<2>)
http -> Saw end of headers... ( *http api.del.icio.us:443*<2>)
http -> url-http-parse-response called in ( *http api.del.icio.us:443*<2>)
http -> Saw chunked encoding.
http -> Calling initial chunked-encoding for extra data at end of headers
http -> Reading chunk 0 (338 480 142)
http -> Saw start of chunk 1 (length=78, start=339
http -> Reading chunk 1 (338 480 142)
http -> Got to the end of chunk #1!
http -> Saw start of chunk 2 (length=0, start=417
http -> Saw end of stream chunk!
http -> Saw end of trailers...
http -> Marking connection as free: api.del.icio.us:443 #<process api.del.icio.us>
http -> url-http-parse-headers called in ( *http api.del.icio.us:443*<2>)
http -> url-http-parse-response called in ( *http api.del.icio.us:443*<2>)
http -> Parsed HTTP headers: class=2 status=200
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: api.del.icio.us:443 #<process api.del.icio.us>
http -> Activating callback in buffer ( *http api.del.icio.us:443*<2>)
retrieval -> Synchronous fetching done (#<buffer  *http api.del.icio.us:443*<2>>)
http -> Spinning waiting for headers...
http -> url-http-end-of-document-sentinel in buffer ( *http api.del.icio.us:443*<2>)
http -> Marking connection as free: api.del.icio.us:443 #<process api.del.icio.us>
http -> url-http-parse-headers called in ( *http api.del.icio.us:443*<2>)
http -> url-http-parse-response called in ( *http api.del.icio.us:443*<2>)
http -> Parsed HTTP headers: class=2 status=200
http -> Finished parsing HTTP headers: t
http -> Marking connection as free: api.del.icio.us:443 #<process api.del.icio.us>
http -> Activating callback in buffer ( *http api.del.icio.us:443*<2>)
retrieval -> Synchronous fetching done (#<buffer  *http api.del.icio.us:443*<2>>)



If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/home/happy/src/emacs/emacs-cvs/etc/DEBUG for instructions.


In GNU Emacs 23.0.50.1 (i686-pc-linux-gnu, GTK+ Version 2.10.13, multi-tty)
 of 2007-08-31 on stone
Important settings:
  value of $LC_ALL: C
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: C
  locale-coding-system: nil
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
RET C-x b SPC * h t TAB RET ESC < C-x k RET C-x k SPC 
* h TAB RET ESC [ A ESC [ A C-@ ESC [ B ESC [ B ESC 
[ B ESC x r e p o r TAB RET

Recent messages:
Contacting host: api.del.icio.us:443
Opening TLS connection to `api.del.icio.us'...
Opening TLS connection with `gnutls-cli -p %p %h'...done
Opening TLS connection to `api.del.icio.us'...done
Mark set [2 times]
Loading emacsbug...
Loading regexp-opt...done
Loading emacsbug...done
Loading dabbrev...done
Scanning for dabbrevs...99%