Re: Release 1.2.1: status

[zimoun <zimon.toutoune@gmail.com>]

Hi Luis,

Thanks for testings and reporting.


On Fri, 19 Mar 2021 at 22:26, Luis Felipe <luis.felipe.la@protonmail.com> wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, March 18, 2021 2:28 PM, zimoun <zimon.toutoune@gmail.com> wrote:
>
> [...]
>
>>     We are still missing a good story to monitor what is archived on
>>     Software Heritage and what is not. Because for now there is rate limit,
>>     I am not able to automate… Well, it is a long WIP. :-)
>>
>>     To help and avoid this rate limit, if all of us simply run:
>>
>>     for pkg in $(guix package -I | cut -f1);
>>     do
>>     guix lint -c archival $pkg
>>     done
>>
>>     for all our profiles, it will ensure at least a coverage for the
>>     packages using git-fetch that we individually care. Note the option
>>     --manifest for “guix lint” is missing… should happen soon if no one
>>     beats me. :-)
>
> I tried this with my user profile and the script hit the Software
> heritage limit after reporting the status of 33 packages (most of
> which are not archived). So, for the rest of the packages, you are
> asked to try again later (I have 95 packages in my profile).

There is 2 rate limit: one for saving and one for requesting.

Each time you do “guix lint -c archival <foo>”, Guix requests to SWH via
their API [1] if the package is already in.  AFAIR, it is 120 requests
per hour.

Then if it is not, Guix saves to SWH via their API.  And this rate is
very low, maybe 10 per hour.  Well, if I remember correctly.


1: <https://archive.softwareheritage.org/api/>

> Is "guix lint -c archival $pkg" supposed to poke Software Heritage to
> archive the $pkg if it is not archived yet? I ask because I ran the
> script later and I got the same output from the first run, i.e.,
> packages reported to be missing from Software Heritage were still
> reported as such, instead of being planned for archive.

Currently, the request/save via “guix lint -c archival” is not optimal.
For instance, the source of the package “libvirt“ is url-fetch so
requesting for it is not necessary because it cannot be saved via their
API.  And I not remember exactly how the ‘tarball’ request is counted.

BTW, the packages using ’url-fetch’ should be ingested by SWH via their
nixguix loader reading the sources.json [2].  And for example:

--8<---------------cut here---------------start------------->8---
$ guix lint -c archival libvirt
gnu/packages/virtualization.scm:1070:5: libvirt@5.8.0: source not archived on Software Heritage

$ guix download -H sha256 -f base64 https://libvirt.org/sources/libvirt-5.8.0.tar.xz

Starting download of /tmp/guix-file.XWwdFj
From https://libvirt.org/sources/libvirt-5.8.0.tar.xz...
 libvirt-5.8.0.tar.xz  12.5MiB        678KiB/s 00:19 [##################] 100.0%
/gnu/store/1pgi1bl8p7jv2mhk83kv7raak2b4k1w5-libvirt-5.8.0.tar.xz
4jMoKJsYve3B6Wb2wmQCspgxScZg7YvVLNpv6rDCDFU=
--8<---------------cut here---------------end--------------->8---

and in the same time, the sources.json contains: 

--8<---------------cut here---------------start------------->8---
    {
      "type": "url",
      "urls": [
        "https://libvirt.org/sources/libvirt-5.8.0.tar.xz"
      ],
      "integrity": "sha256-4jMoKJsYve3B6Wb2wmQCspgxScZg7YvVLNpv6rDCDFU="
    },
--8<---------------cut here---------------end--------------->8---

Well, 2 possible explanations:

 1) The tarball is not in SWH; because their loader fails on it or for
 whatever else reasons
 
 2) Or the tarball is archived by SWH but they use another hashing
 (SWH-ID) than the NAR.  Well, with the information in the package, Guix
 is not able to ask to SWH with the correct hash.  That’s the main
 motivation behind disarchive [3].

Last, Guix is not able to deal with hg-fetch or svn-fetch.  In this
message [4] and the 2 follow-up in the thread, there is some explanation
to implement ‘lookup-subversion-revision’ in (guix swh). Maybe for the
next release on Nov. ;-)

Well, the dance SWH needs some love. :-)


Thanks for trying!  It really helps to have this kind of feedback.



2: <http://guix.gnu.org/sources.json>
3: <https://git.ngyro.com/disarchive>
4: <http://issues.guix.gnu.org/43442#9>


> Also, I got a backtrace when checking icecat:
>
> ★★★★★★★★★★★★★★★
> Backtrace:cecat@78.8.0-guix0-preview1 [archival]...

[...]

> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
> In procedure bv-length: Wrong type argument in position 1 (expecting bytevector): #f
> ★★★★★★★★★★★★★★★

Indeed, there is a bug.  Because the source of ’icecat’ raises a case
that is not handled by ’check-archival’ in (guix lint).

Basically in the snippet:

--8<---------------cut here---------------start------------->8---
                 (match (lookup-content (content-hash-value hash)
                                        (symbol->string
                                         (content-hash-algorithm hash)))
--8<---------------cut here---------------end--------------->8---

’lookup-content’ expect a bytevector for ’content-hash’ and in the case
of ’icecat’, it returns #f.  Then raises the backtrace.  Could you open
a bug report for that?  Just to not forget to fix it. :-)


For the record, compare ’icecat’ with ’hello’:

--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.5
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(guix packages)
scheme@(guix-user)> ,use(guix swh)
scheme@(guix-user)> ,use(gnu packages gnuzilla)
scheme@(guix-user)> (content-hash-value (origin-hash (package-source icecat)))
$1 = #f
scheme@(guix-user)> (lookup-content (content-hash-value (origin-hash (package-source icecat))) "sha256")
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
In procedure bv-length: Wrong type argument in position 1 (expecting bytevector): #f

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
scheme@(guix-user) [1]> ,q

scheme@(guix-user)> ,use(gnu packages base)
scheme@(guix-user)> (content-hash-value (origin-hash (package-source hello)))
$3 = #vu8(49 224 102 19 122 150 38 118 232 159 105 209 182 83 130 222 149 167 239 125 145 75 140 185 86 244 30 167 46 15 81 107)
scheme@(guix-user)> (lookup-content (content-hash-value (origin-hash (package-source hello))) "sha256")
$2 = #<<content> checksums: (("sha1" . #vu8(247 190 191 111 156 98 162 41 94 136 159 102 224 92 233 191 174 217 172 227)) ("blake2s256" . #vu8(4 255 253 50 132 65 210 22 201 36 146 173 114 211 115 136 216 199 120 137 136 11 6 145 81 41 135 134 253 72 216 137)) ("sha1_git" . #vu8(202 230 179 60 195 63 170 253 45 107 216 108 107 66 115 249 51 140 105 194)) ("sha256" . #vu8(49 224 102 19 122 150 38 118 232 159 105 209 182 83 130 222 149 167 239 125 145 75 140 185 86 244 30 167 46 15 81 107))) data-url: "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/raw/" file-type-url: "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/filetype/" language-url: "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/language/" length: 725946 license-url: "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/license/">
--8<---------------cut here---------------end--------------->8---


Cheers,
simon