Re: url.el blocks Gnus+nnrss

[Klaus Straubinger <KSNetz@Arcor.DE>]

> OT1H, looking at the docstring of url-retrieve, I'd think that it's a
> bug in url-http.el that it sometimes fails to activate the callback.
>
> But, OTHO, looking at the url-http.el code it seems that it purposefully
> only activates the callback when the retrieval was successful.
>
> If someone could explain to me which it is, that would help.

I can't explain it.

Unfortunately, there is no good method to always react in the right
way, especially for these connections without specified Content-Length
where the only indicator is that the connection has been closed
already. As far as I know, there is no mechanism in Emacs that triggers
a function at connection-state changes. And how would you then classify
whether the retrieval has been successful?

I think most of the complexity of the URL package comes from the fact
that it implements asynchronous fetching of data. This feature is not
necessary for my usage pattern, but obviously the author thought the
complexity worthwile.

> To debug it, could you try to replace the (setq retrieval-done t) in
> your patch with (debug). Also do (setq url-debug '(http)).

OK, I have done that.

> When the debugger gets called, take a look at the *URL-DEBUG* buffer
> (and send it here).

Here it is:

    http -> Cleaning up dead process: proxy:8080 #<process proxy>
    http -> Cleaning up dead process: proxy:8080 #<process proxy>
    http -> Contacting host: proxy:8080
    http -> Marking connection as busy: proxy:8080 #<process proxy>
    http -> Request is: 
    GET http://www.chesscenter.com/twic/twic.html HTTP/1.1
    MIME-Version: 1.0
    Connection: close
    Extension: Security/Digest Security/SSL
    Host: www.chesscenter.com
    Accept-charset: utf-8;q=1, iso-8859-15;q=0.5, iso-8859-1;q=0.5, iso-8859-2;q=0.5, iso-8859-3;q=0.5, iso-8859-4;q=0.5, iso-8859-5;q=0.5, iso-8859-7;q=0.5, iso-8859-8;q=0.5, iso-8859-9;q=0.5, gb2312;q=0.5, euc-jp;q=0.5, euc-kr;q=0.5, iso-8859-14;q=0.5, big5;q=0.5, iso-2022-jp;q=0.5, koi8-r;q=0.5, shift_jis;q=0.5, viscii;q=0.5, hz-gb-2312;q=0.5, iso-2022-cn-ext;q=0.5, iso-2022-cn;q=0.5, iso-2022-jp-2;q=0.5, iso-2022-kr;q=0.5
    Accept-language: de-DE, de, en-GB, en, it-IT, it, es-ES, es
    Accept: */*
    User-Agent: Emacs-W3/Exp URL/Emacs
    

    http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process proxy>'
    http -> url-http-wait-for-headers-change-function ( *http proxy:8080*<3>)
    http -> Saw end of headers... ( *http proxy:8080*<3>)
    http -> url-http-parse-response called in ( *http proxy:8080*<3>)
    http -> No content-length, being dumb.
    http -> Calling after change function `url-http-simple-after-change-function' for `#<process proxy>'
    [...
     many identical lines cut
     ...]
    http -> Calling after change function `url-http-simple-after-change-function' for `#<process proxy>'
    http -> url-http-end-of-document-sentinel in buffer ( *http proxy:8080*<3>)
    http -> Marking connection as free: proxy:8080 #<process proxy>
    http -> url-http-parse-headers called in ( *http proxy:8080*<3>)
    http -> url-http-parse-response called in ( *http proxy:8080*<3>)
    http -> Parsed HTTP headers: class=2 status=200
    http -> Finished parsing HTTP headers: t
    http -> Marking connection as free: proxy:8080 #<process proxy>
    http -> Activating callback in buffer ( *http proxy:8080*<3>)

The debugger backtrace looks like this, exactly as one would expect:

    (if (eq (process-status proc) (quote closed)) (debug) (if (accept-process-output proc) nil (setq proc ...)))
    (while (not (symbol-value --retrieval-done--30970)) (url-debug (quote retrieval) "Spinning in url-retrieve-synchronously: %S (%S)" (symbol-value --retrieval-done--30970) (symbol-value --asynch-buffer--30971)) (if (eq ... ...) (debug) (if ... nil ...)))
    (if (null proc) nil (while (not ...) (url-debug ... "Spinning in url-retrieve-synchronously: %S (%S)" ... ...) (if ... ... ...)))
    (let ((proc ...)) (if (null proc) nil (while ... ... ...)))
    (progn (set --asynch-buffer--30971 (url-retrieve url ...)) (let (...) (if ... nil ...)) (symbol-value --asynch-buffer--30971))
    (let ((--retrieval-done--30970 ...) (--asynch-buffer--30971 ...)) (setf (symbol-value --retrieval-done--30970) nil (symbol-value --asynch-buffer--30971) nil) (progn (set --asynch-buffer--30971 ...) (let ... ...) (symbol-value --asynch-buffer--30971)))
    (lexical-let ((retrieval-done nil) (asynch-buffer nil)) (setq asynch-buffer (url-retrieve url ...)) (let (...) (if ... nil ...)) asynch-buffer)
    url-retrieve-synchronously("http://WWW.ChessCenter.Com/twic/twic.html")
    eval((url-retrieve-synchronously "http://WWW.ChessCenter.Com/twic/twic.html"))
    eval-expression((url-retrieve-synchronously "http://WWW.ChessCenter.Com/twic/twic.html") nil)
    call-interactively(eval-expression)


By the way, I have observed also hangs at redirects, where another
url-retrieve is called when the old connection could still receive
data. (accept-process-output url-http-process) directly before the new
url-retrieve did help, but I have no clue why.

> not and null are synonyms.  When to use which is a question of taste, but
> I tend to use `not' when applied to something that I consider as a boolean
> value, whereas I tend to use `null' when applied to something that can be
> nil or anything else (a list, for example).
> So in the above case, I think `not' is preferable.

That sounds very reasonable.

-- 
Klaus Straubinger