Downloading with Emacs

Downloading with Emacs

[Lennart Borgman]

I have been testing a bit downloading things from the Internet with 
Emacs with the url package. However I am not able to understand how this 
should be done. Does anyone have examples of how to download binary files?

Another question: What about proxy servers with the url elisp package?

Re: Downloading with Emacs

[Paul Pogonyshev]

Lennart Borgman wrote:
> I have been testing a bit downloading things from the Internet with
> Emacs with the url package. However I am not able to understand how this
> should be done. Does anyone have examples of how to download binary files?

You can have a look at Wikipedia mode (https://gna.org/projects/wikipedia-el/).
It actually doesn't use binary files, but as far as I understand URL retrieves
things in binary (since I manage to use `decode-coding-string' with the
expected result.)  To finish file downloading, you just need to remove HTTP
headers and save the buffer in a file, I guess.

> Another question: What about proxy servers with the url elisp package?

See customization.  Proxy can be configured there.

Paul

Re: Downloading with Emacs

[Lennart Borgman]

Paul Pogonyshev wrote:

>Lennart Borgman wrote:
>  
>
>>I have been testing a bit downloading things from the Internet with
>>Emacs with the url package. However I am not able to understand how this
>>should be done. Does anyone have examples of how to download binary files?
>>    
>>
>
>You can have a look at Wikipedia mode (https://gna.org/projects/wikipedia-el/).
>It actually doesn't use binary files, but as far as I understand URL retrieves
>things in binary (since I manage to use `decode-coding-string' with the
>expected result.)  To finish file downloading, you just need to remove HTTP
>headers and save the buffer in a file, I guess.
>
Thanks Paul, that helped even though I have been using just 
`url-retrieve' like you do in wikipedia.el for my tests. I was wondering 
if there were any "url-download functions".

I believe I will write a little package for downloading then, it seems 
useful.

Re: Downloading with Emacs

[Lennart Borgman]

Lennart Borgman wrote:

> I believe I will write a little package for downloading then, it seems 
> useful.

I called the package url-dl.el. If anyone is interested you can find it 
on http://ourcomments.org/Emacs/DL/inst/.

Re: Downloading with Emacs

[Lennart Borgman]

Robert J. Chassell wrote:

>I just tried
>
>    (url-dl-download "http://purl.org/harder/dpans.html" 
>                     "/usr/local/src/" 
>                     nil 
>                     "Common_Lisp.texi")
>
>and received this error message:
>
>    error in process filter: progn: 
>    Opening output file: no such file or directory, 
>    /home/bob/emacs-21/http:/www.ifa.au.dk/~harder/dpans.html-h.txt
>
>I also tried the command replacing the nil argument with "" but the
>expression also failed.
>
>What should I do?
>  
>
Could not avoid testing. I believe it is a bug in the url.el -libraries. 
There is a callback telling a redirection was done. I recieve this 
instead of the one I should get when the download has finished.

Is there anyone who understands those libraries?

Re: Downloading with Emacs - bug in url-http.el?

[Lennart Borgman]

Robert J. Chassell wrote:

>I just tried
>
>    (url-dl-download "http://purl.org/harder/dpans.html" 
>                     "/usr/local/src/" 
>                     nil 
>                     "Common_Lisp.texi")
>
>and received this error message:
>
>    error in process filter: progn: 
>    Opening output file: no such file or directory, 
>    /home/bob/emacs-21/http:/www.ifa.au.dk/~harder/dpans.html-h.txt
>
>I also tried the command replacing the nil argument with "" but the
>expression also failed.
>  
>
I have tried to find the cause if this. It seems like the callback 
functions from url-retrieve does not work as it should. I get callbacks 
from redirections though I believe it is not documented that it should 
behave this way.

I have implemented a workaround. This workaround unfortunately means you 
can not have more than one download active (and it is not foolproof).

Re: Downloading with Emacs

[Richard M. Stallman]

    Could not avoid testing. I believe it is a bug in the url.el -libraries. 
    There is a callback telling a redirection was done. I recieve this 
    instead of the one I should get when the download has finished.

    Is there anyone who understands those libraries?

I don't think anyone here really understands them.
If you want to start learning about them, so you can maintain
them, that would be very useful.

Re: Downloading with Emacs

[Stefan Monnier]

>> I just tried
>> 
>> (url-dl-download "http://purl.org/harder/dpans.html"
>> "/usr/local/src/"                     nil
>> "Common_Lisp.texi")
>> 
>> and received this error message:
>> 
>> error in process filter: progn:    Opening output file: no such file or
>> directory,
>> /home/bob/emacs-21/http:/www.ifa.au.dk/~harder/dpans.html-h.txt
>> 
>> I also tried the command replacing the nil argument with "" but the
>> expression also failed.
>> 
>> What should I do?
>> 
>> 
> Could not avoid testing. I believe it is a bug in the
> url.el -libraries. There is a callback telling a redirection was
> done. I recieve this instead of the one I should get when the download
> has finished.

I assume you use url-retrieve-synchronously or url-insert-file-contents.
For user-level use, I use url-handler-mode so that I can just do
C-x C-f http://foo/bar RET

Note that a "fix" in the area of http redirection was installed recently,
so maybe the bug you see was fixed, or maybe the fix introduced another bug
you're experiencing.

> Is there anyone who understands those libraries?

I don't understand URL, really, but I have worked on it a tiny bit.


        Stefan