Guix Data Services: whishlist about SWH

Guix Data Services: whishlist about SWH

[zimoun]

Hi,

Recently, the source of package was missing and so "guix time-machine"
was broken [1]. It is not the first time that it appears [2]; patches
are coming... :-)

Using the Software Heritage (SWH) API [3], does it seems a good idea
to add SWH coverage somewhere in the Guix Data Services?

I do not remember where the Reproducibility chart is located, but a
chart there could be added, telling how many packages are already
archived in SWH and how many not yet.

And for example, on the webpage about the history of the package [4],
some information about SWH could be added.

What do you think?


[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39575
[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28659
[3] https://archive.softwareheritage.org/api/
[4] http://data.guix.gnu.org/repository/1/branch/master/package/harfbuzz


(Note that on [4] the version 2.4.0 which is the culprit is not shown,
is it expected?)

All the best,
simon

Re: Guix Data Services: whishlist about SWH

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Recently, the source of package was missing and so "guix time-machine"
> was broken [1]. It is not the first time that it appears [2]; patches
> are coming... :-)
>
> Using the Software Heritage (SWH) API [3], does it seems a good idea
> to add SWH coverage somewhere in the Guix Data Services?
>
> I do not remember where the Reproducibility chart is located, but a
> chart there could be added, telling how many packages are already
> archived in SWH and how many not yet.
>
> And for example, on the webpage about the history of the package [4],
> some information about SWH could be added.
>
> What do you think?

I think that would be good data to have in the Guix Data Service.

I'm not sure quite how this data would fit in, I guess it would be
similar to the data for nar files, but more limited to fixed output
derivations.

I think the first step towards this would be to experiment with fetching
data from the Software Heritage API. Do you know how you'd fetch data
about the output from a fixed output derivation (like harfbuzz)?

http://data.guix.gnu.org/gnu/store/53mfydv96wfwg2fxjyyml44v7v2y84z3-harfbuzz-2.5.3.tar.xz

> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39575
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28659
> [3] https://archive.softwareheritage.org/api/
> [4] http://data.guix.gnu.org/repository/1/branch/master/package/harfbuzz
>
>
> (Note that on [4] the version 2.4.0 which is the culprit is not shown,
> is it expected?)
>
> All the best,
> simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Guix Data Services: whishlist about SWH

[zimoun]

Hi Chris,

On Sun, 16 Feb 2020 at 10:00, Christopher Baines <mail@cbaines.net> wrote:

> I think the first step towards this would be to experiment with fetching
> data from the Software Heritage API. Do you know how you'd fetch data
> about the output from a fixed output derivation (like harfbuzz)?
>
> http://data.guix.gnu.org/gnu/store/53mfydv96wfwg2fxjyyml44v7v2y84z3-harfbuzz-2.5.3.tar.xz

Right now, I do not know. :-)
But I think Guix is already able to fallback to SWH... Well, I am
going to investigate. :-)

Ludo, an input? ;-)


Cheers,
simon

Re: Guix Data Services: whishlist about SWH

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> zimoun <zimon.toutoune@gmail.com> writes:

[...]

>> Using the Software Heritage (SWH) API [3], does it seems a good idea
>> to add SWH coverage somewhere in the Guix Data Services?

That’s a great idea!  Maybe it’s not a good idea to store that info in
the database of the Data Service though.  (Web browsers could query the
SWH API by themselves actually, though it’d be nice to have a JS-free
variant.)

> I think the first step towards this would be to experiment with fetching
> data from the Software Heritage API. Do you know how you'd fetch data
> about the output from a fixed output derivation (like harfbuzz)?

The ‘archival’ linter does that.  See also the examples at:

  https://guix.gnu.org/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/

Ludo’.

Re: Guix Data Services: whishlist about SWH

[zimoun]

Hi,

On Mon, 24 Feb 2020 at 18:10, Ludovic Courtès <ludo@gnu.org> wrote:

> >> Using the Software Heritage (SWH) API [3], does it seems a good idea
> >> to add SWH coverage somewhere in the Guix Data Services?
>
> That’s a great idea!  Maybe it’s not a good idea to store that info in
> the database of the Data Service though.  (Web browsers could query the
> SWH API by themselves actually, though it’d be nice to have a JS-free
> variant.)

It is out of my skills. :-)

What I have in mind is:
 1. Green/red button like SWH stamp! :-) Green for archived in SWH and
red for not yet, and maybe orange for scheduled.
 2. A chart for the coverage.

Well, the point 1. works at the package level and should be done
entirely inside the webbrowser (modulo JS-free).
However, point 2. cannot be checked on the fly, IMHO and so needs to
be stored somewhere in the Data Service.

I have no idea where to look to for example start something about the
point 1. Clue accepted. ;-)


> > I think the first step towards this would be to experiment with fetching
> > data from the Software Heritage API. Do you know how you'd fetch data
> > about the output from a fixed output derivation (like harfbuzz)?
>
> The ‘archival’ linter does that.  See also the examples at:
>
>   https://guix.gnu.org/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/

Thank you for the pointer. I will try to find my way in the linter code. :-)


Thinking about connections between Guix/Nix and SWH and discussing
with lewo, it could be also nice to "push" for each evaluation of
Cuirass all our upstream sources to the SWH listener. Well, the SWH
listener is still a work-in-progress but currently it accepts
something in the format of 'sources.json'. A patch implementing that
is pending [1] and in the near future (after soonish the patch's
review) this 'sources.json' will be produced by the Website... Hum?
The question is then: what will produce this 'sources.json' file?
Cuirass or Data Services?

If I understand Clément's concern about Cuirass, he is not in favour
to add too much Guix only features to Cuirass. So Cuirass does not
seem the right place.
Well, the Data Service could do that. For each evaluation, it produces
this 'sources.json' file and it serves it a a place that the SWH
listener could ingest and so schedule some fetching.
There is still questions to minimize the resource consumption (more on
SWH side than on our side), but that another story.

In this case, Guix will be more robust, especially about time travelling.

What do you think?

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#29
[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39547#35


All the best,
simon