Re: more url-utils?

[Ted Zlatanov <tzz@lifelogs.com>]

On Tue, 17 May 2011 00:04:50 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Ah, OK.  So my proposal does not fit because you want backwards
>> compatibility.

SM> I think that's right, although I'm not completely clear on what is
SM> exactly your proposal.

This is my proposal, after thinking about it for a bit.

Current top-level API:

Dynamic binding: `url-request-data', `url-request-method' and
`url-request-extra-headers' according to the docstring; also according
to a comment: url-standalone-mode, url-gateway-unplugged,
w3-honor-stylesheets, url-confirmation-func, url-cookie-multiple-line,
url-cookie-{{,secure-}storage,confirmation}.

(defun url-retrieve (url callback &optional cbargs silent) ...)
(defun url-retrieve-synchronously (url) ...)

Both of these result in a buffer containing

"[HEADERS]

[BODY]"

Proposed API (based on code review throughout Emacs and my feeling for
what would be "natural" in ELisp):

;;; buffer-local variables: url-retrieved-ok url-headers-alist url-headers-as-string
(defmacro with-url-contents-buffer (url url-options-alist callback-closure &rest body)
...)

1) Synchronous usage with errors:

(with-url-contents-buffer "http://host" '((url-request-method "POST")) nil
  (message "this will run SYNCHRONOUSLY since the callback-closure is nil")
  (message "this could throw an error")
  (message "Header x is %s" (assq 'x url-headers-alist))
  (message "Header y is %s" (assoc "y" url-headers-alist))
  (message "The headers as a string are %s" url-headers-as-string)
  (when url-retrieved-ok (message "The retrieval went fine"))
  (message "data is %s" (buffer-string))

2) Synchronous usage without errors:

(with-url-contents-buffer "http://host" '((url-request-method "POST")) 'noerror
  (message "this will run SYNCHRONOUSLY since the callback-closure is nil")
  (message "this will not throw an error"))

3) Asynchronous usage with a callback (`url-headers-alist',
`url-headers-as-string', and `url-retrieved-ok' are still available):

(with-url-contents-buffer "http://host" '((url-request-method "POST")) callback-closure
  (message "this will run AFTER the retrieval and the callback-closure are done"))

4) Asynchronous usage with no callback (`url-headers-alist',
`url-headers-as-string', and `url-retrieved-ok' are still available):

(with-url-contents-buffer "http://host" '((url-request-method "POST")) t
  (message "this will run AFTER the retrieval is done, there is no callback"))

Implicitly we don't have an asynchronous mode that throws errors.

I also think we should have an incremental mode where each retrieved
chunk is given to the callback in the buffer as it arrives.  That would
let us handle large downloads better.  The API would not change, though,
only the `url-options-alist' would have a `url-incremental' key and the
callback would look at the buffer-local variable
`url-incremental-byte-offset'.

SM> If you can provide a patch that give us a clean new API together with
SM> some backward compatibility layer, that would be great.

After we've settled on the API, please.  I think the above may be too
different for many :)

Ted