Function to download a URL and return it as a string

Function to download a URL and return it as a string

[Sean McAfee]

I recently wanted to be able to download a document from a URL and
return it as a string.  I couldn't find a function that did that
precisely, but I was able to construct my own:

(defun download (url)
  (with-current-buffer (url-retrieve-synchronously url)
    (prog2
        (when (not (search-forward-regexp "^$" nil t))
          (error "Unable to locate downloaded data"))
        (buffer-substring (1+ (point)) (point-max))
      (kill-buffer))))

This seems rather busy for such a basic-seeming operation--in
particular, having to take care to delete the buffer created by
url-retrieve-synchronously.  Is there a better way to do it?

(On the other hand, this way I get to use prog2, which I almost never
do.)

RE: Function to download a URL and return it as a string

[Drew Adams]

> I recently wanted to be able to download a document from a URL and
> return it as a string.  I couldn't find a function that did that
> precisely, but I was able to construct my own:
> 
> (defun download (url)
>   (with-current-buffer (url-retrieve-synchronously url)
>     (prog2
>         (when (not (search-forward-regexp "^$" nil t))
>           (error "Unable to locate downloaded data"))
>         (buffer-substring (1+ (point)) (point-max))
>       (kill-buffer))))
> 
> This seems rather busy for such a basic-seeming operation--in
> particular, having to take care to delete the buffer created by
> url-retrieve-synchronously.  Is there a better way to do it?

Dunno anything about whether there is a better way, but consider using
`unwind-protect', with `kill-buffer' as your cleanup part.  And to play safe,
pass the actual buffer (returned from `url-*') as arg to `kill-buffer'.

Using `unwind-protect' makes sure the buffer is killed when you're done, no
matter what.

Re: Function to download a URL and return it as a string

[Pascal J. Bourguignon]

Sean McAfee <eefacm@gmail.com> writes:

> I recently wanted to be able to download a document from a URL and
> return it as a string.  I couldn't find a function that did that
> precisely, but I was able to construct my own:
>
> (defun download (url)
>   (with-current-buffer (url-retrieve-synchronously url)
>     (prog2
>         (when (not (search-forward-regexp "^$" nil t))
>           (error "Unable to locate downloaded data"))
>         (buffer-substring (1+ (point)) (point-max))
>       (kill-buffer))))
>
> This seems rather busy for such a basic-seeming operation--in
> particular, having to take care to delete the buffer created by
> url-retrieve-synchronously.  Is there a better way to do it?
>
> (On the other hand, this way I get to use prog2, which I almost never
> do.)

Notice that you cannot just receive the bytes of the ressource.  At the
very least, you also need the Content-Type:.  So don't feel this header
is a bad thing.

I would have used (search-forward "\n\n"): 

(defun download (url)
  (with-current-buffer (url-retrieve-synchronously url)
     (prog1 (buffer-substring (or (search-forward "\n\n" nil t) (point-min)) 
                              (point-max))
       (kill-buffer))))
  
-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: Function to download a URL and return it as a string

[jun net]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

There is a function I can't remember exactly. Maybe with-temp-string? Like
with-current-buffer, but makes all output to a temp string.
在 2012-12-9 下午1:20，"Sean McAfee" <eefacm@gmail.com>写道：
>
> I recently wanted to be able to download a document from a URL and
> return it as a string.  I couldn't find a function that did that
> precisely, but I was able to construct my own:
>
> (defun download (url)
>   (with-current-buffer (url-retrieve-synchronously url)
>     (prog2
>         (when (not (search-forward-regexp "^$" nil t))
>           (error "Unable to locate downloaded data"))
>         (buffer-substring (1+ (point)) (point-max))
>       (kill-buffer))))
>
> This seems rather busy for such a basic-seeming operation--in
> particular, having to take care to delete the buffer created by
> url-retrieve-synchronously.  Is there a better way to do it?
>
> (On the other hand, this way I get to use prog2, which I almost never
> do.)

[-- Attachment #2: Type: text/html, Size: 1307 bytes --]

Re: Function to download a URL and return it as a string

[Jambunathan K]

May be you can steal some stuff from 

    M-x find-library RET package.el

specifically

    C-h f package--with-work-buffer

Or 

You can file an enhancement request so that some of this stuff could be
moved out of package.el to someother library.

ps: I know nothing about handling HTTP requests.  Just pitching in with
few pennies.

Sean McAfee <eefacm@gmail.com> writes:

> I recently wanted to be able to download a document from a URL and
> return it as a string.  I couldn't find a function that did that
> precisely, but I was able to construct my own:
>
> (defun download (url)
>   (with-current-buffer (url-retrieve-synchronously url)
>     (prog2
>         (when (not (search-forward-regexp "^$" nil t))
>           (error "Unable to locate downloaded data"))
>         (buffer-substring (1+ (point)) (point-max))
>       (kill-buffer))))
>
> This seems rather busy for such a basic-seeming operation--in
> particular, having to take care to delete the buffer created by
> url-retrieve-synchronously.  Is there a better way to do it?
>
> (On the other hand, this way I get to use prog2, which I almost never
> do.)
>

-- 

Re: Function to download a URL and return it as a string

[Sean McAfee]

"Drew Adams" <drew.adams@oracle.com> writes:
>> (defun download (url)
>>   (with-current-buffer (url-retrieve-synchronously url)
>>     (prog2
>>         (when (not (search-forward-regexp "^$" nil t))
>>           (error "Unable to locate downloaded data"))
>>         (buffer-substring (1+ (point)) (point-max))
>>       (kill-buffer))))

> Dunno anything about whether there is a better way, but consider using
> `unwind-protect', with `kill-buffer' as your cleanup part.

My original implementation used unwind-protect, but on reflection it
seemed quite unnecessary when there was no way for the protected form to
ever raise an exception.

I suppose there's a future-proofing argument to be made, but it's hard
to see this code evolving to the point where an exception ever could
occur.