all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* url-fetch with referer
@ 2019-05-23 18:11 Pierre Neidhardt
  2019-05-28 15:06 ` Ludovic Courtès
  0 siblings, 1 reply; 5+ messages in thread
From: Pierre Neidhardt @ 2019-05-23 18:11 UTC (permalink / raw)
  To: Guix-devel

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

Hi,

I've hit a source archive which won't get downloaded properly with
url-fetch.

This is because the hosting site requires a "referer", and url-fetch
does not seem to provide any.

For instance

  wget https://foo/bar.zip

fails, while

  wget --referer https://foo https://foo/bar.zip

works.

Can we set a referer in url-fetch, or at least provide an option?  Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: url-fetch with referer
  2019-05-23 18:11 url-fetch with referer Pierre Neidhardt
@ 2019-05-28 15:06 ` Ludovic Courtès
  2019-05-29 14:28   ` Pierre Neidhardt
  0 siblings, 1 reply; 5+ messages in thread
From: Ludovic Courtès @ 2019-05-28 15:06 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: Guix-devel

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> For instance
>
>   wget https://foo/bar.zip
>
> fails, while
>
>   wget --referer https://foo https://foo/bar.zip
>
> works.

I say that it’s a buggy web site.  :-)

More generally, we’ve added a couple of HTTP headers for issues that
were relatively common.  We could easily fix this one, but maybe you
could first talk to the webmasters, because requiring ‘Referer’ sounds
obnoxious; WDYT?

(Note that in the meantime you can always work around the problem by
writing your own fixed-output derivation.)

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: url-fetch with referer
  2019-05-28 15:06 ` Ludovic Courtès
@ 2019-05-29 14:28   ` Pierre Neidhardt
  2019-05-31 21:39     ` Ludovic Courtès
  0 siblings, 1 reply; 5+ messages in thread
From: Pierre Neidhardt @ 2019-05-29 14:28 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: Guix-devel

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> More generally, we’ve added a couple of HTTP headers for issues that
> were relatively common.  We could easily fix this one, but maybe you
> could first talk to the webmasters, because requiring ‘Referer’ sounds
> obnoxious; WDYT?

So I did for 1 website and the answer was that this is on purpose, to
"avoid linking directly to the archive from a different website."  A
rather delusional enforcement, but hey, if we can't convince the webmasters...

How would we add the relevant HTTP header then?

> (Note that in the meantime you can always work around the problem by
> writing your own fixed-output derivation.)

I don't understand, can you explain?


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: url-fetch with referer
  2019-05-29 14:28   ` Pierre Neidhardt
@ 2019-05-31 21:39     ` Ludovic Courtès
  2019-06-01 10:20       ` Pierre Neidhardt
  0 siblings, 1 reply; 5+ messages in thread
From: Ludovic Courtès @ 2019-05-31 21:39 UTC (permalink / raw)
  To: Pierre Neidhardt; +Cc: Guix-devel

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> More generally, we’ve added a couple of HTTP headers for issues that
>> were relatively common.  We could easily fix this one, but maybe you
>> could first talk to the webmasters, because requiring ‘Referer’ sounds
>> obnoxious; WDYT?
>
> So I did for 1 website and the answer was that this is on purpose, to
> "avoid linking directly to the archive from a different website."  A
> rather delusional enforcement, but hey, if we can't convince the webmasters...
>
> How would we add the relevant HTTP header then?

See ‘http-fetch’ in (guix build download).

(It would be more convincing if this is for a package to be included in
Guix proper, hint hint. :-))

>> (Note that in the meantime you can always work around the problem by
>> writing your own fixed-output derivation.)
>
> I don't understand, can you explain?

You could write something like:

--8<---------------cut here---------------start------------->8---
(define* (my-url-fetch url hash-algo hash
                       #:optional name
                       #:key (system (%current-system))
                       (guile (default-guile)))
  (define file-name
    (match url
      ((head _ ...)
       (basename head))
      (_
       (basename url))))

  (let ((uri (and (string? url) (string->uri url))))
    (gexp->derivation name
                      #~(begin
                          (use-modules (web client))

                          (http-get … #:headers …)
                          …)
                      ;; …
                      #:hash hash
                      #:hash-algo hash-algo)))
--8<---------------cut here---------------end--------------->8---

HTH!

Ludo’.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: url-fetch with referer
  2019-05-31 21:39     ` Ludovic Courtès
@ 2019-06-01 10:20       ` Pierre Neidhardt
  0 siblings, 0 replies; 5+ messages in thread
From: Pierre Neidhardt @ 2019-06-01 10:20 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: Guix-devel

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> (It would be more convincing if this is for a package to be included in
> Guix proper, hint hint. :-))

Hint taken, but hey, the world being what it is...  One battle at a time! :D

> You could write something like:
>
> --8<---------------cut here---------------start------------->8---
> (define* (my-url-fetch url hash-algo hash
> ...
> --8<---------------cut here---------------end--------------->8---

Makes total sense, thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2019-06-01 10:34 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-05-23 18:11 url-fetch with referer Pierre Neidhardt
2019-05-28 15:06 ` Ludovic Courtès
2019-05-29 14:28   ` Pierre Neidhardt
2019-05-31 21:39     ` Ludovic Courtès
2019-06-01 10:20       ` Pierre Neidhardt

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.