Downloading a file from a URL from the web with http-get

Downloading a file from a URL from the web with http-get

[白熊]

Hello everybody:

I'm learning Guile. I'd like to write a utility, which will enable me upon user input download program sources from the web, i.e. not a web-spider, but download discrete URL's and save them to disk.

I've looked at:

http://lists.gnu.org/archive/html/guile-user/2012-04/msg00032.html
http://lists.gnu.org/archive/html/guile-user/2012-05/msg00005.html

so the possibility to do, seems to be there.

Specifically, let's say, I want to download:

http://ftp.gnu.org/pub/gnu/guile/guile-2.0.11.tar.xz

and save it to ~/

How should I modify the following code to do this?:

(use-modules ((web uri)    #:select (string->uri))
               ((web client) #:select (http-get)))
  (call-with-values (lambda ()
                      (http-get
                       (string->uri "http://www.gnu.org/software/guile/")))
    (lambda (res-headers res-body)
      (display res-body)
      (newline)))

i.e. not display the body, but save it to file? Or is the above only applicable to text responses - webpages, and not for binary files?

Thanks very much for any help.
--
白熊

Re: Downloading a file from a URL from the web with http-get

[Dmitry Bogatov]

[-- Attachment #1: Type: text/plain, Size: 1448 bytes --]

* 白熊 <guile-user_gnu.org@sumou.com> [2014-08-04 12:12:03+0400]
> Hello everybody:
>
> I'm learning Guile. I'd like to write a utility, which will enable me
> upon user input download program sources from the web, i.e. not a
> web-spider, but download discrete URL's and save them to disk.
>
> I've looked at:
>
> http://lists.gnu.org/archive/html/guile-user/2012-04/msg00032.html
> http://lists.gnu.org/archive/html/guile-user/2012-05/msg00005.html
>
> so the possibility to do, seems to be there.
>
> Specifically, let's say, I want to download:
>
> http://ftp.gnu.org/pub/gnu/guile/guile-2.0.11.tar.xz
>
> and save it to ~/
>
> How should I modify the following code to do this?:
It seems that `http-get` do not manage SSL, and
GNU site 302 to https version (or is it just my proxy config?)

So here is example for other url:

	(use-modules ((web uri) #:select (string->uri))
		     ((web client) #:select (http-get)))
	(use-modules (rnrs io ports))

	(define *url*
	  "http://hackage.haskell.org/package/HaTeX-2.1.3/HaTeX-2.1.3.tar.gz")

	(call-with-values
	    (lambda () (http-get (string->uri *url*)))
	  (lambda (res-headers res-body)
	    (with-output-to-file "some.tar.gz"
	      (lambda () (put-bytevector (current-output-port) res-body))
	      #:binary #t)))

--
Best regards, Dmitry Bogatov <KAction@gnu.org>,
Free Software supporter, esperantisto and netiquette guardian.
GPG: 54B7F00D

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Downloading a file from a URL from the web with http-get

[白熊]

On 2014年8月4日 20:57:57 GMT+04:00, Dmitry Bogatov <KAction@gnu.org> wrote:

>It seems that `http-get` do not manage SSL, and
>GNU site 302 to https version (or is it just my proxy config?)

Thank you, Dmitry. 

It seems to be your setup, as your code example works for me also on the GNU site. 

>So here is example for other url:
>
>	(use-modules ((web uri) #:select (string->uri))
>		     ((web client) #:select (http-get)))
>	(use-modules (rnrs io ports))
>
>	(define *url*
>	  "http://hackage.haskell.org/package/HaTeX-2.1.3/HaTeX-2.1.3.tar.gz")
>
>	(call-with-values
>	    (lambda () (http-get (string->uri *url*)))
>	  (lambda (res-headers res-body)
>	    (with-output-to-file "some.tar.gz"
>	      (lambda () (put-bytevector (current-output-port) res-body))
>	      #:binary #t)))

Thank you for this, it's exactly what I was looking for. 

Also, is there a way to determine the exact file size, so I could check before downloading, if the file has already been downloaded, i.e. same size on disk as on the web, or if the prior download failed, i.e. smaller size on disk, than the web? 

I thought the headers would have the size info of the tarball, but I see no such information in res-headers... 

But surely, the information must be available as any download tool like wget or a browser is able to determine the download file size. 

How can I do this? 
-- 
白熊

Re: Downloading a file from a URL from the web with http-get

[Dmitry Bogatov]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

* 白熊 <guile-user_gnu.org@sumou.com> [2014-08-05 03:56:03+0400]
> Also, is there a way to determine the exact file size, so I could
> check before downloading, if the file has already been downloaded,
> i.e. same size on disk as on the web, or if the prior download failed,
> i.e. smaller size on disk, than the web?
>
> I thought the headers would have the size info of the tarball, but I see no such information in res-headers...
>
> But surely, the information must be available as any download tool
> like wget or a browser is able to determine the download file size.

Seems that wget also do not have this information.

wget -S http://hackage.haskell.org/package/HaTeX-2.1.3/HaTeX-2.1.3.tar.gz
--2014-08-05 11:27:59--  http://hackage.haskell.org/package/HaTeX-2.1.3/HaTeX-2.1.3.tar.gz
Resolving hackage.haskell.org (hackage.haskell.org)... 88.198.224.242, 88.198.224.242
Connecting to hackage.haskell.org (hackage.haskell.org)|88.198.224.242|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.6.0
  Date: Mon, 04 Aug 2014 16:53:31 GMT
  Content-Type: application/x-gzip
  Content-MD5: c4d3cf03316068d1ef0799024f6a1dfd
  ETag: "c4d3cf03316068d1ef0799024f6a1dfd"
  Last-Modified: Fri, 22 Oct 2010 21:18:12 GMT
  Age: 52477
  Transfer-Encoding: chunked
  Connection: keep-alive
Length: unspecified [application/x-gzip]
Saving to: `HaTeX-2.1.3.tar.gz'

Look, it says, that length is unspecified. My best bet is use
#:streaming? flag to `http-get` to examine headers before downloading, and if
I decide to download, get data from port via `get-bytevector` function
family. This way you also can examine first several bytes before decide,
whether file is needed, if headers information is not sufficient.

--
Best regards, Dmitry Bogatov <KAction@gnu.org>,
Free Software supporter, esperantisto and netiquette guardian.
GPG: 54B7F00D

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: Downloading a file from a URL from the web with http-get

[白熊]

On 2014年8月4日 18:57:57 CEST, Dmitry Bogatov <KAction@gnu.org> wrote:
>So here is example for other url:
>
>	(use-modules ((web uri) #:select (string->uri))
>		     ((web client) #:select (http-get)))
>	(use-modules (rnrs io ports))
>
>	(define *url*
>	  "http://hackage.haskell.org/package/HaTeX-2.1.3/HaTeX-2.1.3.tar.gz")
>
>	(call-with-values
>	    (lambda () (http-get (string->uri *url*)))
>	  (lambda (res-headers res-body)
>	    (with-output-to-file "some.tar.gz"
>	      (lambda () (put-bytevector (current-output-port) res-body))
>	      #:binary #t)))

Is there a way to use similar syntax to download ftp files? 
I can successfully download only http:// files this way. 
--
白熊

Re: Downloading a file from a URL from the web with http-get

[Mike Gran]

> On Friday, August 15, 2014 11:16 AM, 白熊 <guile-user_gnu.org@sumou.com> wrote:
> Is there a way to use similar syntax to download ftp files? 
> I can successfully download only http:// files this way. 


I don't know if anyone has a pure Scheme code solution to handle FTP.

There was once a binding to the libcurl library that could do FTP and TFTP,
but it would requiring compiling and installing a binary extension.  So, it
is not very convenient.  But, there's some info here:

http://www.lonelycactus.com/guile-curl.html

I haven't tried it lately, though.

-Mike


> 
> --
> 白熊
>

Re: Downloading a file from a URL from the web with http-get

[Mario Domenech Goulart]

Hi,

On Fri, 15 Aug 2014 13:46:46 -0700 Mike Gran <spk121@yahoo.com> wrote:

>> On Friday, August 15, 2014 11:16 AM, 白熊 <guile-user_gnu.org@sumou.com> wrote:
>> Is there a way to use similar syntax to download ftp files? 
>> I can successfully download only http:// files this way. 
>
> I don't know if anyone has a pure Scheme code solution to handle FTP.
>
> There was once a binding to the libcurl library that could do FTP and TFTP,
> but it would requiring compiling and installing a binary extension.  So, it
> is not very convenient.  But, there's some info here:
>
> http://www.lonelycactus.com/guile-curl.html
>
> I haven't tried it lately, though.

There's a pure Scheme implementation for CHICKEN [1].  Maybe it can be
ported to Guile.  I haven't used it, though, so I can't tell you about
its working status.

[1] http://wiki.call-cc.org/egg/ftp (code here:
http://bugs.call-cc.org/browser/release/4/ftp/trunk/ftp.scm).

Best wishes.
Mario
-- 
http://parenteses.org/mario

Re: Downloading a file from a URL from the web with http-get

[Ludovic Courtès]

FWIW, Guix also has an FTP client in (guix ftp-client), and it’s
independent from the rest of Guix:

  http://git.savannah.gnu.org/cgit/guix.git/tree/guix/ftp-client.scm

It also uses (web client) extensively, with the GnuTLS Guile bindings
for HTTPS:

  http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/download.scm#n115

Thanks,
Ludo’.