issue with SWH ratelimit (authorization via HTTP header)

issue with SWH ratelimit (authorization via HTTP header)

[zimoun]

Hi,

I am puzzled by the module ’(web client)’.

SWH allows authorization via token which sets the ratelimit to 1200
instead of 120.

<https://archive.softwareheritage.org/api/>

But I fails to use it with Guile.  Well, first thing first:

--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.7
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(web client)

scheme@(guix-user)> (http-request "https://archive.softwareheritage.org/api/1/stat/counters/" #:method 'GET)

$1 = #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers: ((date . #<date nanosecond: 0 second: 44 minute: 19 hour: 18 day: 14 month: 10 year: 2021 zone-offset: 0>) (server . "gunicorn/19.9.0") (content-type application/json) (vary accept cookie accept-encoding) (allow GET OPTIONS HEAD OPTIONS) (x-frame-options . "SAMEORIGIN") (via "1.1 archive.softwareheritage.org") (x-varnish . "35616175 35955724") (age . 129) (via "1.1 varnish (Varnish/6.1)") (strict-transport-security . "max-age=15768000;") (accept-ranges bytes) (content-length . 124) (connection close)) port: #<closed: file 7f6f16d40c40>>
$2 = #vu8(123 34 111 114 105 103 105 110 34 58 49 54 54 49 55 49 49 54 50 44 34 114 101 118 105 115 105 111 110 34 58 50 52 49 53 54 49 49 54 56 51 44 34 99 111 110 116 101 110 116 34 58 49 49 51 57 53 48 51 54 48 55 55 44 34 100 105 114 101 99 116 111 114 121 34 58 57 53 48 50 49 56 57 52 48 56 44 34 114 101 108 101 97 115 101 34 58 50 49 50 48 53 52 48 51 44 34 112 101 114 115 111 110 34 58 52 53 56 55 54 56 51 53 125)

scheme@(guix-user)> (http-request "https://archive.softwareheritage.org/api/1/stat/counters/" #:method 'POST)

$3 = #<<response> version: (1 . 1) code: 405 reason-phrase: "Method Not Allowed" headers: ((date . #<date nanosecond: 0 second: 41 minute: 22 hour: 18 day: 14 month: 10 year: 2021 zone-offset: 0>) (server . "gunicorn/19.9.0") (content-type application/json) (vary accept cookie) (allow GET OPTIONS HEAD OPTIONS) (x-ratelimit-limit . "120") (x-ratelimit-remaining . "117") (x-ratelimit-reset . "1634235776") (x-frame-options . "SAMEORIGIN") (content-length . 72) (via "1.1 archive.softwareheritage.org") (x-varnish . "35616222") (age . 0) (via "1.1 varnish (Varnish/6.1)") (strict-transport-security . "max-age=15768000;") (connection close)) port: #<closed: file 7f6f150250e0>>
$4 = #vu8(123 34 101 120 99 101 112 116 105 111 110 34 58 34 77 101 116 104 111 100 78 111 116 65 108 108 111 119 101 100 34 44 34 114 101 97 115 111 110 34 58 34 77 101 116 104 111 100 32 92 34 80 79 83 84 92 34 32 110 111 116 32 97 108 108 111 119 101 100 46 34 125)
--8<---------------cut here---------------end--------------->8---

Here, it works but as you can see, the X-RateLimit is at 120.

Next step, define the token from SWH.  And it fails.

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (define token 'eyJhbGc…)

scheme@(guix-user)> (http-request "https://archive.softwareheritage.org/api/1/stat/counters/" #:method 'GET #:headers `((authorization . (bearer ,token))))

$5 = #<<response> version: (1 . 1) code: 403 reason-phrase: "Forbidden" headers: ((date . #<date nanosecond: 0 second: 42 minute: 24 hour: 18 day: 14 month: 10 year: 2021 zone-offset: 0>) (server . "gunicorn/19.9.0") (content-type application/json) (vary accept cookie) (allow GET OPTIONS HEAD OPTIONS) (x-frame-options . "SAMEORIGIN") (content-length . 104) (via "1.1 archive.softwareheritage.org") (x-varnish . "35741654") (age . 0) (via "1.1 varnish (Varnish/6.1)") (strict-transport-security . "max-age=15768000;") (connection close)) port: #<closed: file 7f6f153ce230>>
$6 = #vu8(123 34 101 120 99 101 112 116 105 111 110 34 58 34 65 117 116 104 101 110 116 105 99 97 116 105 111 110 70 97 105 108 101 100 34 44 34 114 101 97 115 111 110 34 58 34 73 110 118 97 108 105 100 32 111 114 32 117 110 115 117 112 112 111 114 116 101 100 32 72 84 84 80 32 97 117 116 104 111 114 105 122 97 116 105 111 110 32 116 121 112 101 32 40 98 101 97 114 101 114 41 46 34 125)

scheme@(guix-user)> (http-request "https://archive.softwareheritage.org/api/1/stat/counters/" #:method 'POST #:headers `((authorization . (bearer ,token))))

$7 = #<<response> version: (1 . 1) code: 403 reason-phrase: "Forbidden" headers: ((date . #<date nanosecond: 0 second: 53 minute: 24 hour: 18 day: 14 month: 10 year: 2021 zone-offset: 0>) (server . "gunicorn/19.9.0") (content-type application/json) (vary accept cookie) (allow GET OPTIONS HEAD OPTIONS) (x-frame-options . "SAMEORIGIN") (content-length . 104) (via "1.1 archive.softwareheritage.org") (x-varnish . "36086556") (age . 0) (via "1.1 varnish (Varnish/6.1)") (strict-transport-security . "max-age=15768000;") (connection close)) port: #<closed: file 7f6f1681daf0>>
$8 = #vu8(123 34 101 120 99 101 112 116 105 111 110 34 58 34 65 117 116 104 101 110 116 105 99 97 116 105 111 110 70 97 105 108 101 100 34 44 34 114 101 97 115 111 110 34 58 34 73 110 118 97 108 105 100 32 111 114 32 117 110 115 117 112 112 111 114 116 101 100 32 72 84 84 80 32 97 117 116 104 111 114 105 122 97 116 105 111 110 32 116 121 112 101 32 40 98 101 97 114 101 114 41 46 34 125)
--8<---------------cut here---------------end--------------->8---

The most probable explanations is that the header is badly formed.  How
to do it?  Because it works using ’curl’.

--8<---------------cut here---------------start------------->8---
$  curl -i https://archive.softwareheritage.org/api/1/stat/counters/
HTTP/1.1 200 OK
Date: Thu, 14 Oct 2021 18:27:48 GMT
Server: gunicorn/19.9.0
Content-Type: application/json
Vary: Accept,Cookie,Accept-Encoding
Allow: GET, OPTIONS, HEAD, OPTIONS
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 116
X-RateLimit-Reset: 1634236081
X-Frame-Options: SAMEORIGIN
Via: 1.1 archive.softwareheritage.org
X-Varnish: 35551144
Age: 0
Via: 1.1 varnish (Varnish/6.1)
Strict-Transport-Security: max-age=15768000;
Accept-Ranges: bytes
Content-Length: 124
Connection: keep-alive

{"origin":166171162,"revision":2415611683,"content":11395036077,"directory":9502189408,"release":21205686,"person":45876835}

$  curl -i -H "Authorization: Bearer ${TOKEN}" https://archive.softwareheritage.org/api/1/stat/counters/
HTTP/1.1 200 OK
Date: Thu, 14 Oct 2021 18:27:53 GMT
Server: gunicorn/19.9.0
Content-Type: application/json
Vary: Accept,Cookie,Accept-Encoding
Allow: GET, OPTIONS, HEAD, OPTIONS
X-RateLimit-Limit: 1200
X-RateLimit-Remaining: 1195
X-RateLimit-Reset: 1634236075
X-Frame-Options: SAMEORIGIN
Content-Length: 124
Via: 1.1 archive.softwareheritage.org
X-Varnish: 35897944
Age: 0
Via: 1.1 varnish (Varnish/6.1)
Strict-Transport-Security: max-age=15768000;
Accept-Ranges: bytes
Connection: keep-alive

{"origin":166171162,"revision":2415611683,"content":11395036077,"directory":9502189408,"release":21205686,"person":45876835}
--8<---------------cut here---------------end--------------->8---


It would help for checking what is inside SWH.

Thanks for any tips.


All the best,
simon

Re: issue with SWH ratelimit (authorization via HTTP header)

[zimoun]

Hi,

Thanks to Ricardo on #guix-hpc, the issue is:

    `((authorization . (Bearer ,token))))

and not ’bearer’.  Maybe the process can help.

It is misleading because the documentation [1] says to use:

--8<---------------cut here---------------start------------->8---
Enter `,help' for help.
scheme@(guix-user)> ,use(web http)
scheme@(guix-user)> (parse-header 'authorization "Bearer eyJh…")
$1 = (bearer eyJh…)
--8<---------------cut here---------------end--------------->8---

But it is not what

    (http-request URL #:method 'POST #:headers `((authorization . (bearer ,token))))

returns.

Let open a terminal and run ’emacs-simple-httpd’ as server.  The buffer
*httpd* lists what is happening.  In another terminal, let run the
request above.  Tadam!

--8<---------------cut here---------------start------------->8---
(request
 (date "Thu, 14 Oct 2021 18:41:51 GMT")
 (address "127.0.0.1")
 (get "/")
 (headers
  ("POST" "/" "HTTP/1.1")
  ("Host" "localhost:8080")
  ("Connection" "close")
  ("Authorization" "Bearer eyJh…$
  ("Content" ""))
--8<---------------cut here---------------end--------------->8---

Now, with the correct header, it works:

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,use(web client)
scheme@(guix-user)> (http-request "https://archive.softwareheritage.org/api/1/stat/counters/" #:method 'POST #:headers `((authorization . (Bearer ,token))))
$2 = #<<response> version: (1 . 1) code: 405 reason-phrase: "Method Not Allowed" headers: ((date . #<date nanosecond: 0 second: 2 minute: 59 hour: 18 day: 14 month: 10 year: 2021 zone-offset: 0>) (server . "gunicorn/19.9.0") (content-type application/json) (vary accept cookie) (allow GET OPTIONS HEAD OPTIONS) (x-ratelimit-limit . "1200") (x-ratelimit-remaining . "137") (x-ratelimit-reset . "1634237954") (x-frame-options . "SAMEORIGIN") (content-length . 72) (via "1.1 archive.softwareheritage.org") (x-varnish . "35772607") (age . 0) (via "1.1 varnish (Varnish/6.1)") (strict-transport-security . "max-age=15768000;") (connection close)) port: #<closed: file 7f624e3f51c0>>
$4 = #vu8(123 34 101 120 99 101 112 116 105 111 110 34 58 34 77 101 116 104 111 100 78 111 116 65 108 108 111 119 101 100 34 44 34 114 101 97 115 111 110 34 58 34 77 101 116 104 111 100 32 92 34 80 79 83 84 92 34 32 110 111 116 32 97 108 108 111 119 101 100 46 34 125)
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

1: <https://www.gnu.org/software/guile/manual/html_node/HTTP-Headers.html>

Re: issue with SWH ratelimit (authorization via HTTP header)

[Maxim Cournoyer]

Hello,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> Thanks to Ricardo on #guix-hpc, the issue is:
>
>     `((authorization . (Bearer ,token))))
>
> and not ’bearer’.  Maybe the process can help.
>
> It is misleading because the documentation [1] says to use:
>
> Enter `,help' for help.
> scheme@(guix-user)> ,use(web http)
> scheme@(guix-user)> (parse-header 'authorization "Bearer eyJh…")
> $1 = (bearer eyJh…)
>
>
> But it is not what
>
>     (http-request URL #:method 'POST #:headers `((authorization . (bearer ,token))))
>
> returns.
>
> Let open a terminal and run ’emacs-simple-httpd’ as server.  The buffer
> *httpd* lists what is happening.  In another terminal, let run the
> request above.  Tadam!
>
> (request
>  (date "Thu, 14 Oct 2021 18:41:51 GMT")
>  (address "127.0.0.1")
>  (get "/")
>  (headers
>   ("POST" "/" "HTTP/1.1")
>   ("Host" "localhost:8080")
>   ("Connection" "close")
>   ("Authorization" "Bearer eyJh…$
>   ("Content" ""))
>
>
> Now, with the correct header, it works:
>
> scheme@(guix-user)> ,use(web client)
> scheme@(guix-user)> (http-request "https://archive.softwareheritage.org/api/1/stat/counters/" #:method 'POST #:headers `((authorization . (Bearer ,token))))
> $2 = #<<response> version: (1 . 1) code: 405 reason-phrase: "Method Not Allowed" headers: ((date . #<date nanosecond: 0 second: 2 minute: 59 hour: 18 day: 14 month: 10 year: 2021 zone-offset: 0>) (server . "gunicorn/19.9.0") (content-type application/json) (vary accept cookie) (allow GET OPTIONS HEAD OPTIONS) (x-ratelimit-limit . "1200") (x-ratelimit-remaining . "137") (x-ratelimit-reset . "1634237954") (x-frame-options . "SAMEORIGIN") (content-length . 72) (via "1.1 archive.softwareheritage.org") (x-varnish . "35772607") (age . 0) (via "1.1 varnish (Varnish/6.1)") (strict-transport-security . "max-age=15768000;") (connection close)) port: #<closed: file 7f624e3f51c0>>
> $4 = #vu8(123 34 101 120 99 101 112 116 105 111 110 34 58 34 77 101 116 104 111 100 78 111 116 65 108 108 111 119 101 100 34 44 34 114 101 97 115 111 110 34 58 34 77 101 116 104 111 100 32 92 34 80 79 83 84 92 34 32 110 111 116 32 97 108 108 111 119 101 100 46 34 125)

Cool debugging session!

Thanks for sharing,

Maxim