Disarchive and SHA

Disarchive and SHA

[zimoun]

Hi,

Trying to investigate why, for instance,

--8<---------------cut here---------------start------------->8---
$ guix lint -c archival zabbix-agentd
gnu/packages/monitoring.scm:167:5: zabbix-agentd@5.2.6: Disarchive entry refers to non-existent SWH directory 'e664cd5e820df2a194a5c6a64f12161480331b4b'
--8<---------------cut here---------------end--------------->8---

I notice something annoying.  For representing SHA-256, Guix uses
’nix-base32’ and Disarchive uses ’base16’.

Therefore, from the source,

--8<---------------cut here---------------start------------->8---
(define-public zabbix-agentd
  (package
    (name "zabbix-agentd")
    (version "5.2.6")
    (source
     (origin
       (method url-fetch)
       (uri (string-append
             "https://cdn.zabbix.com/zabbix/sources/stable/"
             (version-major+minor version) "/zabbix-" version ".tar.gz"))
       (sha256
        (base32 "100n1rv7r4pqagxxifzpcza5bhrr2fklzx7gndxwiyq4597p1jvn"))))
--8<---------------cut here---------------end--------------->8---

it does not match the URL:

<http://disarchive.guix.gnu.org/sha256/100n1rv7r4pqagxxifzpcza5bhrr2fklzx7gndxwiyq4597p1jvn>

and instead, one has to convert to ’base16’.  I am not aware of any tool
to ease this transformation.  Maybe, we could add an option to “guix
hash”.  WDYT?


Using ’base16’, the URL matches and the file is downloaded.  Neat!

<http://disarchive.guix.gnu.org/sha256/76cb704f2a04fbc87bb3eff44fa71339c355d467f7bbd8fb53f8927c760e1680>



Along the process, I also notice,

--8<---------------cut here---------------start------------->8---
$ guix download https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz

Starting download of /tmp/guix-file.rcYxyF
From https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz...
download failed "https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz" 404 "Not Found"

Starting download of /tmp/guix-file.rcYxyF
From https://web.archive.org/web/20211020115955/https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz...
following redirection to `https://web.archive.org/web/20210410075108/https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz'...
 …2.6.tar.gz                                                        350KiB/s 00:57 | 19.6MiB transferred
/gnu/store/3w8mp25gyrkq3dngaw27kvqaggrx9qp0-zabbix-5.2.6.tar.gz
100n1rv7r4pqagxxifzpcza5bhrr2fklzx7gndxwiyq4597p1jvn
--8<---------------cut here---------------end--------------->8---

And this fallback is not done by the file ’sources.json’.

--8<---------------cut here---------------start------------->8---
$ wget https://guix.gnu.org/sources.json
$ cat sources.json | jq | grep zabbix
      "git_url": "https://github.com/lukecyca/pyzabbix",
        "https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz"
      "git_url": "https://github.com/unioslo/zabbix-cli",
        "https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz"
--8<---------------cut here---------------end--------------->8---

That’s why «Disarchive entry refers to non-existent SWH directory».


Help welcome for improving this ’sources.json’. :-)  Especially, turning
the current way using the website builder into derivation-style usable
by the CI.


Cheers,
simon

Re: Disarchive and SHA

[zimoun]

Hi,

On Wed, 20 Oct 2021 at 12:23, zimoun <zimon.toutoune@gmail.com> wrote:

> --8<---------------cut here---------------start------------->8---
> (define-public zabbix-agentd
[...]
>        (sha256
>         (base32 "100n1rv7r4pqagxxifzpcza5bhrr2fklzx7gndxwiyq4597p1jvn"))))
> --8<---------------cut here---------------end--------------->8---
>
> it does not match the URL:
>
> <http://disarchive.guix.gnu.org/sha256/100n1rv7r4pqagxxifzpcza5bhrr2fklzx7gndxwiyq4597p1jvn>
>
> and instead, one has to convert to ’base16’.  I am not aware of any tool
> to ease this transformation.  Maybe, we could add an option to “guix
> hash”.  WDYT?

For the interested reader, the trick is to use...

--8<---------------cut here---------------start------------->8---
$ guix hash -f base16 $(guix build zabbix-agentd -S)
76cb704f2a04fbc87bb3eff44fa71339c355d467f7bbd8fb53f8927c760e1680
--8<---------------cut here---------------end--------------->8---

> Using ’base16’, the URL matches and the file is downloaded.  Neat!
>
> <http://disarchive.guix.gnu.org/sha256/76cb704f2a04fbc87bb3eff44fa71339c355d467f7bbd8fb53f8927c760e1680>

...but it must download.  Instead, I propose the patch#51307 which
convert without downloading.  WDYT?

1: <http://issues.guix.gnu.org/51307>


Cheers,
simon

Re: Disarchive and SHA

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Along the process, I also notice,
>
> $ guix download https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz
>
> Starting download of /tmp/guix-file.rcYxyF
> From https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz...
> download failed "https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz" 404 "Not Found"
>
> Starting download of /tmp/guix-file.rcYxyF
> From https://web.archive.org/web/20211020115955/https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz...
> following redirection to `https://web.archive.org/web/20210410075108/https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz'...
>  …2.6.tar.gz                                                        350KiB/s 00:57 | 19.6MiB transferred
> /gnu/store/3w8mp25gyrkq3dngaw27kvqaggrx9qp0-zabbix-5.2.6.tar.gz
> 100n1rv7r4pqagxxifzpcza5bhrr2fklzx7gndxwiyq4597p1jvn
>
>
> And this fallback is not done by the file ’sources.json’.
>
> $ wget https://guix.gnu.org/sources.json
> $ cat sources.json | jq | grep zabbix
>       "git_url": "https://github.com/lukecyca/pyzabbix",
>         "https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz"
>       "git_url": "https://github.com/unioslo/zabbix-cli",
>         "https://cdn.zabbix.com/zabbix/sources/stable/5.2/zabbix-5.2.6.tar.gz"
>
> That’s why «Disarchive entry refers to non-existent SWH directory».

However, some time ago, the zabbix.com URL was 200-OK, and at that point
SWH would have ingested it, no?

Or were we unlucky and the URL was already 404 by the time SWH read its
first ‘sources.json’?

> Help welcome for improving this ’sources.json’. :-)  Especially, turning
> the current way using the website builder into derivation-style usable
> by the CI.

Yes, it’s probably a good idea to eventually turn ‘sources.json’ into a
‘computed-file’, and make a one-element manifest under etc/.

Thanks,
Ludo’.

Re: Disarchive and SHA

[zimoun]

Hi,

On Thu, 21 Oct 2021 at 22:28, Ludovic Courtès <ludo@gnu.org> wrote:

>> That’s why «Disarchive entry refers to non-existent SWH directory».
>
> However, some time ago, the zabbix.com URL was 200-OK, and at that point
> SWH would have ingested it, no?

Timothy pointed [1] then click to «Show all visits».

1: <https://archive.softwareheritage.org/browse/origin/visits/?origin_url=https://guix.gnu.org/sources.json>


> Or were we unlucky and the URL was already 404 by the time SWH read its
> first ‘sources.json’?

Well, I do not know how to check.  Somehow, the ’sources.json’ should
contains the fixed-outputs substitutes in addition to the upstream
content.  And probably from both build farms.

It is something I wanted to do but I never took the time to complete
because it was not clear at the time how to compute the hash in
address.  Now fixed-output is cleared for me. :-)

>> Help welcome for improving this ’sources.json’. :-)  Especially, turning
>> the current way using the website builder into derivation-style usable
>> by the CI.
>
> Yes, it’s probably a good idea to eventually turn ‘sources.json’ into a
> ‘computed-file’, and make a one-element manifest under etc/.

It is not clear how to deal with packages into a ’computed-file’.  For
instance, etc/disarchive-manifest.scm create one store item per package
(somehow) then build a directory union.  Here the union is a file.
Well, it is not clear for me how to process.


Cheers,
simon