[bug#51216] [PATCH 0/1] swh: Allows token from Software Heritage authentication service.

[bug#51216] [PATCH 0/1] swh: Allows token from Software Heritage authentication service.

[zimoun]

Hi,

This patch allows to use a token from SWH and change the rate limit when
submitting via the archival checker.  For instance, it reads,

   GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival

and will use the ${TOKEN} associated to the SWH account.  Having an account
enable by default a rate limit to 1200 (instead of 120).  Then, for testing
purpose I get a lot of android packages scheduled after checking many admin.

It is interesting to note lines as:

  Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'

which shows that sources.json is maybe not fully working as expected.
Investigations required.

--8<---------------cut here---------------start------------->8---
sitour@jack-Precision-7820-Tower$  GUIX_SWH_TOKEN=$TOKEN  ./pre-inst-env guix lint -c archival
gnu/packages/accessibility.scm:100:5: brltty@6.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:4330:5: pam-mount@2.18: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:3902:12: launchmon@1.0.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:933:12: net-base@5.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2272:12: testdisk@7.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2148:12: acpica@20210930:
Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
gnu/packages/admin.scm:3029:12: nmap@7.92: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:725:5: pies@1.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2116:5: dmidecode@3.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:1108:5: prips@1.1.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2939:12: munge@0.5.14: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:534:12: facter@4.0.52: scheduled Software Heritage archival
gnu/packages/admin.scm:1137:12: alive@2.0.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2677:5: debops@1.1.0: scheduled Software Heritage archival
checking ansible@4.4.0 [archival]...
gnu/packages/admin.scm:3571:12: neofetch@7.1.0: scheduled Software Heritage archival
gnu/packages/aidc.scm:112:12: qrencode@4.1.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/aidc.scm:174:5: zbar@0.23: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:489:12: arb@2.21.0: scheduled Software Heritage archival
gnu/packages/algebra.scm:1053:14: eigen-for-tensorflow@3.3.5-1.fd6845384b86: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:1104:12: xtensor@0.20.10: scheduled Software Heritage archival
gnu/packages/algebra.scm:1682:11: sollya@7.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:1139:5: gap@4.11.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:957:5: symengine@0.6.0: scheduled Software Heritage archival
gnu/packages/algebra.scm:1605:5: iml@1.0.5: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/android.scm:529:12: android-ext4-utils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:659:12: fastboot@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:744:5: git-repo@2.4.1: scheduled Software Heritage archival
gnu/packages/android.scm:323:12: android-libziparchive@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:581:12: android-f2fs-utils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:405:12: mkbootimg@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:358:12: adb@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:302:12: android-libsparse@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:248:12: android-libcutils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:458:12: android-bionic-uapi@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:619:12: android-libutils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:101:5: android-googletest@1.8.0: scheduled Software Heritage archival
gnu/packages/android.scm:195:12: android-liblog@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/animation.scm:67:5: rlottie@0.2: scheduled Software Heritage archival
gnu/packages/antivirus.scm:48:12: clamav@0.103.3:
Disarchive entry refers to non-existent SWH directory 'b25414c9864a270899ca1ff494e7ba4c437b166d'
gnu/packages/aspell.scm:115:12: aspell-dict-en@2020.12.07-0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:898:5: indi@1.9.1: scheduled Software Heritage archival
gnu/packages/astronomy.scm:239:5: wcslib@7.5: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:298:5: gnuastro@0.16: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:823:5: gpredict@2.2.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:518:5: stuff@1.26.0:
Disarchive entry refers to non-existent SWH directory '128bbe76a82dd0b38b725565ed703a7148257ae0'
gnu/packages/astronomy.scm:216:5: eye@1.4.1:
Disarchive entry refers to non-existent SWH directory '92625e2c6dbe3ad7c4f44a061ada24ce00637087'
gnu/packages/astronomy.scm:377:5: skymaker@3.10.5:
Disarchive entry refers to non-existent SWH directory '6000a273dfff9de62725b53e41562fff711069c1'
gnu/packages/astronomy.scm:278:5: weightwatcher@1.12:
Disarchive entry refers to non-existent SWH directory 'c68ff8714c6fd360a38158f3d8f22e555c061452'
gnu/packages/audio.scm:2941:12: supercollider@3.11.2: scheduled Software Heritage archival
gnu/packages/audio.scm:3131:12: rubberband@1.8.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:5431:6: mda-lv2@1.2.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3906:12: zita-alsa-pcmi@0.3.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2581:12: lilv@0.24.10: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:309:12: libgme@0.6.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4813:5: redkite@1.3.0: scheduled Software Heritage archival
gnu/packages/audio.scm:1908:12: guitarix@0.41.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4657:5: gnaural@20110606: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:1962:2: guitarix-lv2@0.41.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:947:12: azr3@1.2.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4266:12: snd@20.9:
Disarchive entry refers to non-existent SWH directory 'cb52aaa9500df2b674bf7922811deeea1b766139'
gnu/packages/audio.scm:2811:12: patchage@1.0.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:5130:6: lsp-dsp-lib@0.5.11:
Disarchive entry refers to non-existent SWH directory '3e574043a04d77dd7231d23210547c4fe065a40c'
gnu/packages/audio.scm:4114:5: gsm@1.0.19:
Disarchive entry refers to non-existent SWH directory 'aa763150704fe06f34097b38e839409cee52366d'
gnu/packages/audio.scm:4956:6: libaudec@0.2.4: scheduled Software Heritage archival
gnu/packages/audio.scm:3699:5: qsynth@0.5.7:
Disarchive entry refers to non-existent SWH directory '127c0a03c7ccba74870aef7dac36019af35798cc'
gnu/packages/audio.scm:3425:5: wavpack@5.4.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3286:12: timidity++@2.15.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2176:12: jalv@1.6.4: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3802:12: zita-convolver@4.0.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2541:6: audio-to-midi@2020.7: scheduled Software Heritage archival
gnu/packages/audio.scm:563:12: aubio@0.4.9: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3199:12: sratom@0.6.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3226:12: suil@0.10.8: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3853:12: zita-resampler@1.6.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2627:12: lv2@1.18.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3170:12: rtmidi@4.0.0:
Disarchive entry refers to non-existent SWH directory 'd9745f29da983c6ad674871e68ac96362c4f11cc'
gnu/packages/audio.scm:528:12: alsa-modular-synth@2.2.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/augeas.scm:37:12: augeas@1.12.0:
Disarchive entry refers to non-existent SWH directory '7d7ed9f88ee649a90493f54d3988a062c3ddeafb'
gnu/packages/autogen.scm:38:5: autogen@5.18.16: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/autotools.scm:291:5: autoconf-archive@2021.02.19: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/backup.scm:765:12: wimlib@1.13.4: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/backup.scm:192:5: hdup@2.0.14: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/base.scm:587:11: binutils@2.33.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/batik.scm:204:6: java-xmlgraphics-commons@2.6:
Disarchive entry refers to non-existent SWH directory 'f5bd0fe7450175196c57d6f6d5aca8905393e814'
gnu/packages/benchmark.scm:57:12: fio@3.27: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/bioconductor.scm:779:12: r-txdb-hsapiens-ucsc-hg38-knowngene@3.4.6:
Disarchive entry refers to non-existent SWH directory '92bd3b93caa9a4b0840c70ddb96ac75b0684d7ec'
gnu/packages/bioconductor.scm:10755:5: r-timeseriesexperiment@1.10.1: Software Heritage rate limit reached; try again later
--8<---------------cut here---------------end--------------->8---

All the best,
simon


zimoun (1):
  swh: Allows token from Software Heritage authentication service.

 guix/swh.scm | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)


base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58
-- 
2.32.0

[bug#51216] [PATCH 1/1] swh: Allows token from Software Heritage authentication service.

[zimoun]

The token is provided using the environment variable GUIX_SWH_TOKEN.

* guix/swh.scm (swh-token): New variable.
(http-get*, http-post*): Use it.
---
 guix/swh.scm | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/guix/swh.scm b/guix/swh.scm
index 5c41685a24..38a4af723a 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -153,12 +154,30 @@ (define url
       url
       (string-append url "/")))
 
+;; Token from an account to the Software Heritage Authentication service
+;; <https://archive.softwareheritage.org/api/>
+(define swh-token
+  (match (getenv "GUIX_SWH_TOKEN")
+    (#f #f)
+    ((? string-null? s) #f)
+    ((? string? s) (string->symbol s))))
+
 ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would
 ;; be ignored (<https://bugs.gnu.org/40486>).
 (define* (http-get* uri #:rest rest)
-  (apply http-request uri #:method 'GET rest))
+  (apply http-request uri #:method 'GET
+         #:headers
+         (if swh-token
+             `((authorization . (Bearer ,swh-token)))
+             '())
+         rest))
 (define* (http-post* uri #:rest rest)
-  (apply http-request uri #:method 'POST rest))
+  (apply http-request uri #:method 'POST
+         #:headers
+         (if swh-token
+             `((authorization . (Bearer ,swh-token)))
+             '())
+         rest))
 
 (define %date-regexp
   ;; Match strings like "2014-11-17T22:09:38+01:00" or
-- 
2.32.0

[bug#51216] [PATCH 0/1] swh: Allows token from Software Heritage authentication service.

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> This patch allows to use a token from SWH and change the rate limit when
> submitting via the archival checker.  For instance, it reads,
>
>    GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival
>
> and will use the ${TOKEN} associated to the SWH account.  Having an account
> enable by default a rate limit to 1200 (instead of 120).  Then, for testing

Ooh, neat!

> purpose I get a lot of android packages scheduled after checking many admin.

You mean the SWH admins manually scheduled them?

> It is interesting to note lines as:
>
>   Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
>
> which shows that sources.json is maybe not fully working as expected.
> Investigations required.

Weird.  I guess we should pick a few specific cases, make sure they’re
listed in sources.json and they’re not 404, and then ask the SWH team
whether they can check their logs to understand what happened.

Thanks!

Ludo’.

[bug#51216] [PATCH 0/1] swh: Allows token from Software Heritage authentication service.

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> The token is provided using the environment variable GUIX_SWH_TOKEN.
>
> * guix/swh.scm (swh-token): New variable.
> (http-get*, http-post*): Use it.

[...]

> +;; Token from an account to the Software Heritage Authentication service
> +;; <https://archive.softwareheritage.org/api/>
> +(define swh-token
> +  (match (getenv "GUIX_SWH_TOKEN")
> +    (#f #f)
> +    ((? string-null? s) #f)
> +    ((? string? s) (string->symbol s))))

Could you turn it into a SRFI-39 parameter?

>  ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would
>  ;; be ignored (<https://bugs.gnu.org/40486>).
>  (define* (http-get* uri #:rest rest)
> -  (apply http-request uri #:method 'GET rest))
> +  (apply http-request uri #:method 'GET
> +         #:headers
> +         (if swh-token
> +             `((authorization . (Bearer ,swh-token)))
> +             '())
> +         rest))
>  (define* (http-post* uri #:rest rest)
> -  (apply http-request uri #:method 'POST rest))
> +  (apply http-request uri #:method 'POST
> +         #:headers
> +         (if swh-token
> +             `((authorization . (Bearer ,swh-token)))
> +             '())
> +         rest))

These two procedures are meant to go away as soon as Guile > 3.0.2 is
required.

Could you instead pass #:headers in the ‘call’ procedure, and maybe in
‘vault-fetch’ too?

Thanks!

Ludo’.

[bug#51216] [PATCH 0/1] swh: Allows token from Software Heritage authentication service.

[zimoun]

Re,

On Thu, 14 Oct 2021 at 23:44, Ludovic Courtès <ludo@gnu.org> wrote:

>> purpose I get a lot of android packages scheduled after checking many admin.
>
> You mean the SWH admins manually scheduled them?

No no, sorry my bad wording.  I mean “guix lint -c archival” checked all
the 260+ packages in gnu/packages/admin.scm then it checked
gnu/packages/android.scm, found some unarchived and scheduled them.


>> It is interesting to note lines as:
>>
>>   Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
>>
>> which shows that sources.json is maybe not fully working as expected.
>> Investigations required.
>
> Weird.  I guess we should pick a few specific cases, make sure they’re
> listed in sources.json and they’re not 404, and then ask the SWH team
> whether they can check their logs to understand what happened.

Yeah… investigations required. :-)


Cheers,
simon

[bug#51216] [PATCH v2] swh: Allows token from Software Heritage authentication service.

[zimoun]

The token is provided using the environment variable GUIX_SWH_TOKEN.

* guix/swh.scm (%swh-token): New variable.
(call): Use it.
---
 guix/swh.scm | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/guix/swh.scm b/guix/swh.scm
index 5c41685a24..c7c1c873a2 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -136,6 +137,12 @@ (define %verify-swh-certificate?
   ;; Whether to verify the X.509 HTTPS certificate for %SWH-BASE-URL.
   (make-parameter #t))
 
+;; Token from an account to the Software Heritage Authentication service
+;; <https://archive.softwareheritage.org/api/>
+(define %swh-token
+  (make-parameter (and=> (getenv "GUIX_SWH_TOKEN")
+                         string->symbol)))
+
 (define (swh-url path . rest)
   ;; URLs returned by the API may be relative or absolute. This has changed
   ;; without notice before. Handle both cases by detecting whether the path
@@ -246,6 +253,10 @@ (define* (call url decode #:optional (method http-get*)
   (and ((%allow-request?) url method)
        (let*-values (((response port)
                       (method url #:streaming? #t
+                              #:headers
+                              (if (%swh-token)
+                                  `((authorization . (Bearer ,(%swh-token))))
+                                  '())
                               #:verify-certificate?
                               (%verify-swh-certificate?))))
          ;; See <https://archive.softwareheritage.org/api/#rate-limiting>.

base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58
-- 
2.32.0

bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage authentication service.

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> The token is provided using the environment variable GUIX_SWH_TOKEN.
>
> * guix/swh.scm (%swh-token): New variable.
> (call): Use it.

Applied, thanks!  :-)

Ludo'.