Re: url.el blocks Gnus+nnrss

[Stefan Monnier <monnier@iro.umontreal.ca>]

> As far as I know, there is no mechanism in Emacs that triggers
> a function at connection-state changes.

See set-process-sentinel ;-)

> I think most of the complexity of the URL package comes from the fact
> that it implements asynchronous fetching of data. This feature is not
> necessary for my usage pattern, but obviously the author thought the
> complexity worthwile.

Actually, Emacs does not provide any direct support for synchronous
"open-connection/send-data/read-data", the way it does for processes with
`call-process'.  So the author (William Perry) had no choice, really.
Furthermore, given the complexity of HTTP (with authentication and stuff),
it's basically impossible for Emacs to provide a synchronous API that would
be usable for HTTP.

> Here it is:
>     http -> Cleaning up dead process: proxy:8080 #<process proxy>

Hmm... so you're going through a proxy, good to know.

>      many identical lines cut
>      ...]
>     http -> Calling after change function `url-http-simple-after-change-function' for `#<process proxy>'
>     http -> url-http-end-of-document-sentinel in buffer ( *http proxy:8080*<3>)
>     http -> Marking connection as free: proxy:8080 #<process proxy>
>     http -> url-http-parse-headers called in ( *http proxy:8080*<3>)
>     http -> url-http-parse-response called in ( *http proxy:8080*<3>)
>     http -> Parsed HTTP headers: class=2 status=200
>     http -> Finished parsing HTTP headers: t
>     http -> Marking connection as free: proxy:8080 #<process proxy>
>     http -> Activating callback in buffer ( *http proxy:8080*<3>)

So the sentinel is properly called when the connection is closed and it does
activate "the" callback.  Now why didn't the callback set `retrieval-done'?

> The debugger backtrace looks like this, exactly as one would expect:

>     (if (eq (process-status proc) (quote closed)) (debug) (if (accept-process-output proc) nil (setq proc ...)))
>     (while (not (symbol-value --retrieval-done--30970)) (url-debug (quote retrieval) "Spinning in url-retrieve-synchronously: %S (%S)" (symbol-value --retrieval-done--30970) (symbol-value --asynch-buffer--30971)) (if (eq ... ...) (debug) (if ... nil ...)))
>     (if (null proc) nil (while (not ...) (url-debug ... "Spinning in url-retrieve-synchronously: %S (%S)" ... ...) (if ... ... ...)))
>     (let ((proc ...)) (if (null proc) nil (while ... ... ...)))
>     (progn (set --asynch-buffer--30971 (url-retrieve url ...)) (let (...) (if ... nil ...)) (symbol-value --asynch-buffer--30971))
>     (let ((--retrieval-done--30970 ...) (--asynch-buffer--30971 ...)) (setf (symbol-value --retrieval-done--30970) nil (symbol-value --asynch-buffer--30971) nil) (progn (set --asynch-buffer--30971 ...) (let ... ...) (symbol-value --asynch-buffer--30971)))
>     (lexical-let ((retrieval-done nil) (asynch-buffer nil)) (setq asynch-buffer (url-retrieve url ...)) (let (...) (if ... nil ...)) asynch-buffer)
>     url-retrieve-synchronously("http://WWW.ChessCenter.Com/twic/twic.html")
>     eval((url-retrieve-synchronously "http://WWW.ChessCenter.Com/twic/twic.html"))
>     eval-expression((url-retrieve-synchronously "http://WWW.ChessCenter.Com/twic/twic.html") nil)
>     call-interactively(eval-expression)

Could you in the *Backtrace* buffer check the value of the following
expressions (and post them here), using `e':

  proc

  (process-buffer proc)

  (with-current-buffer (process-buffer proc) url-callback-function)

  --retrieval-done--30970

  (symbol-value --retrieval-done--30970)

The "30970" might change from one run to the other, so check the backtrace
to see which number was used that time.

> By the way, I have observed also hangs at redirects, where another
> url-retrieve is called when the old connection could still receive
> data. (accept-process-output url-http-process) directly before the new
> url-retrieve did help, but I have no clue why.

Could you show a patch of the actual change you tried?
Do you men just before `url-retrieve' in url-http-parse-headers
in the lines:

	   (let ((url-request-method url-http-method)
		 (url-request-data url-http-data)
		 (url-request-extra-headers url-http-extra-headers))
	     (url-retrieve redirect-uri url-callback-function
			   url-callback-arguments)
	     (url-mark-buffer-as-dead (current-buffer))))))


-- Stefan