url-fetch with referer

url-fetch with referer

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

Hi,

I've hit a source archive which won't get downloaded properly with
url-fetch.

This is because the hosting site requires a "referer", and url-fetch
does not seem to provide any.

For instance

  wget https://foo/bar.zip

fails, while

  wget --referer https://foo https://foo/bar.zip

works.

Can we set a referer in url-fetch, or at least provide an option?  Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: url-fetch with referer

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> For instance
>
>   wget https://foo/bar.zip
>
> fails, while
>
>   wget --referer https://foo https://foo/bar.zip
>
> works.

I say that it’s a buggy web site.  :-)

More generally, we’ve added a couple of HTTP headers for issues that
were relatively common.  We could easily fix this one, but maybe you
could first talk to the webmasters, because requiring ‘Referer’ sounds
obnoxious; WDYT?

(Note that in the meantime you can always work around the problem by
writing your own fixed-output derivation.)

Thanks,
Ludo’.

Re: url-fetch with referer

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> More generally, we’ve added a couple of HTTP headers for issues that
> were relatively common.  We could easily fix this one, but maybe you
> could first talk to the webmasters, because requiring ‘Referer’ sounds
> obnoxious; WDYT?

So I did for 1 website and the answer was that this is on purpose, to
"avoid linking directly to the archive from a different website."  A
rather delusional enforcement, but hey, if we can't convince the webmasters...

How would we add the relevant HTTP header then?

> (Note that in the meantime you can always work around the problem by
> writing your own fixed-output derivation.)

I don't understand, can you explain?


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: url-fetch with referer

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> More generally, we’ve added a couple of HTTP headers for issues that
>> were relatively common.  We could easily fix this one, but maybe you
>> could first talk to the webmasters, because requiring ‘Referer’ sounds
>> obnoxious; WDYT?
>
> So I did for 1 website and the answer was that this is on purpose, to
> "avoid linking directly to the archive from a different website."  A
> rather delusional enforcement, but hey, if we can't convince the webmasters...
>
> How would we add the relevant HTTP header then?

See ‘http-fetch’ in (guix build download).

(It would be more convincing if this is for a package to be included in
Guix proper, hint hint. :-))

>> (Note that in the meantime you can always work around the problem by
>> writing your own fixed-output derivation.)
>
> I don't understand, can you explain?

You could write something like:

--8<---------------cut here---------------start------------->8---
(define* (my-url-fetch url hash-algo hash
                       #:optional name
                       #:key (system (%current-system))
                       (guile (default-guile)))
  (define file-name
    (match url
      ((head _ ...)
       (basename head))
      (_
       (basename url))))

  (let ((uri (and (string? url) (string->uri url))))
    (gexp->derivation name
                      #~(begin
                          (use-modules (web client))

                          (http-get … #:headers …)
                          …)
                      ;; …
                      #:hash hash
                      #:hash-algo hash-algo)))
--8<---------------cut here---------------end--------------->8---

HTH!

Ludo’.

Re: url-fetch with referer

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> (It would be more convincing if this is for a package to be included in
> Guix proper, hint hint. :-))

Hint taken, but hey, the world being what it is...  One battle at a time! :D

> You could write something like:
>
> --8<---------------cut here---------------start------------->8---
> (define* (my-url-fetch url hash-algo hash
> ...
> --8<---------------cut here---------------end--------------->8---

Makes total sense, thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]