Another update on the Guix Data Service

Another update on the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 5575 bytes --]

Hey,

It's been a while since I send out an update about the Guix Data
Service, so here's a somewhat failed attempt at a short update about
recent progress.

There are formatted [1] and plain views of derivations [2]. The plain
view is constructed from the representation in the database, but it
should match what Guix produces.

1: http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/formatted
2: http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/plain

Channel news entries are recorded in the database, you can see all that
are present in a revision [3], but the comparison page should show new
news entries as well.

3: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news

There's now a page to compare two derivations [4], it's not a
particularly clear comparison yet, but can hopefully be improved in the
future. This is linked to from the derivation history pages [5].

4: http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv&target_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv
5: http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history

There's support for receiving build information from build servers, but
this hasn't been used yet. I'm slowly looking at getting support for
using this in to Cuirass [6]

6: https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00467.html

Information about nars (substitutes) can now be stored ([7] for
example). This data also shows up on a page for derivation outputs for a
revision ([8] for example).

7: http://data.guix.gnu.org/gnu/store/001j0wk7srm7jlnd9lai7p1yzg4bdl3k-dunst-1.4.1
8: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/derivation-outputs

There's a page for builds for a revision [9]. There's also a page for
individual builds [10].

9: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/builds
10: http://data.guix.gnu.org/build-server/2/build?derivation_file_name=/gnu/store/yc0pb7c0c198nb68c5wha41brzvrls01-k3b-19.08.3.drv

The derivation outputs page for a revision shows information about
outputs matching across build servers, but there's also a page about
package reproducibility for a revision [11].

11: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/package-reproducibility

Provide narinfo and nar files for derivations, and the required source
files ([12] for example). This means the Guix Data Service can be used
as a substitute server for derivations, for example [13].

12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo
13: guix build --substitute-urls="http://data.guix.gnu.org https://ci.guix.gnu.org" /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv

Fix generating derivations for foreign architectures. There was an issue
where only the x86_64-linux and i686-linux derivations were properly
loaded, but now that's been fixed.

I added support for loading mbox files a while ago, but I've only now
got around to testing a few older revisions, and now I've loaded the
mbox files for the first 3 months from 2019 so eventually those will be
loaded in.


A few things above were influenced by thinking about reproducible builds
in the run up, during and after the reproducible builds summit at the
start of December. I'm excited that there's some support in the Guix
Data Service for comparing builds across build farms, but there's still
a lot to do in terms of getting every derivation built by at least 2
separate build farms, and then getting that data promptly in to the Guix
Data Service for comparison.

Also at the reproducible builds summit, Ludovic made me aware that you
can tell Guix to build a derivation, and if it doesn't have it locally,
it will attempt to download it from substitute servers. I've now got the
Guix Data Service setup to act as a substitute server for derivations,
which while on it's own isn't particularly exciting (as build farms
already can provide substitutes for derivations), it's now possible to
get the information about derivations within the Guix Data Service, out
of the Guix Data Service in a usable manor. This might have applications
in the future, like speeding up guix pull by providing the derivation
that corresponds to a revision, or providing derivations for a build
farm to build.

Like the example of providing substitutes for derivations, quite a lot
of this work isn't directly impactful (at least yet), but I'm hoping
it's laid the foundations for some more directly impactful stuff.

In particular, I'm still thinking about;

 - Automated code review for Guix patches, which I was working on when I
   started working on the Guix Data Service, and most of the
   functionality can be beneficial there.

 - The Guix Weekly News project I was working on recently uses data from
   the Guix Data Service, and that's what motivated storing the channel
   news entries.

 - Trust in builds, once the data about package build reproducibility is
   more complete, hopefully that will be informative and show where
   build reproducibility issues can be fixed.

 - Improving the speed in which substitutes are made available.

 - Explaining the data that underpins Guix to a technical audience, like
   derivations and how they work.

Do let me know if you have any thoughts or questions, I'm also going to
be around at the Guix Days at the end of January.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Another update on the Guix Data Service

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

Hi Christopher!

Nice update as usual!

A few questions:

> Channel news entries are recorded in the database, you can see all that
> are present in a revision [3], but the comparison page should show new
> news entries as well.
>
> 3: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news

Is there a link to the news page from

http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299

?

>  - Automated code review for Guix patches, which I was working on when I
>    started working on the Guix Data Service, and most of the
>    functionality can be beneficial there.

Some time ago you did some work on Patchwork.

https://patchwork.cbaines.net/project/guix-patches/list/

I haven't followed very closely, so I wonder what your plans are
regarding patchwork and Guix Data Service.

Last, is there any documentation on how to _use_ it?  (I don't mean the setup.)



Cheers, and thanks again with the great work!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Another update on the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> Channel news entries are recorded in the database, you can see all that
>> are present in a revision [3], but the comparison page should show new
>> news entries as well.
>>
>> 3: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news
>
> Is there a link to the news page from
>
> http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299
>
> ?

No, but there probably should be one.

>>  - Automated code review for Guix patches, which I was working on when I
>>    started working on the Guix Data Service, and most of the
>>    functionality can be beneficial there.
>
> Some time ago you did some work on Patchwork.
>
> https://patchwork.cbaines.net/project/guix-patches/list/
>
> I haven't followed very closely, so I wonder what your plans are
> regarding patchwork and Guix Data Service.

So I'd like to resurrect my slightly clunky but somewhat working
Patchwork + Laminar + Git repo + Guix Data Service (+ Cuirass?) setup
for testing patches.

Here, the Guix Data Service would process the Guix revision with the
patches under test having been applied. This would reveal the lint
warning changes. It would also reveal the new derivations, which can be
built to check what the effect of the patches are.

> Last, is there any documentation on how to _use_ it?  (I don't mean the setup.)

So the only documentation currently is the README file, which is
available at http://data.guix.gnu.org/README . I haven't considered
writing standalone documentation yet, but I have been considering
incorporating more in to the web interface.

Is there a specific thing you're interested in/unsure about?

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Another update on the Guix Data Service

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

Hi Christopher,

Thanks for the details!

Christopher Baines <mail@cbaines.net> writes:

> Is there a specific thing you're interested in/unsure about?

Well, mostly the patch continuous integration that you mentioned.

Regarding the documentation, it'd be nice to explain _what_ can be done
with Guix Data Service, since it is rather large already.

Good luck!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Another update on the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> Is there a specific thing you're interested in/unsure about?
>
> Well, mostly the patch continuous integration that you mentioned.

Cool, I'm hoping to have something more concrete by the upcoming Guix
Days!

> Regarding the documentation, it'd be nice to explain _what_ can be done
> with Guix Data Service, since it is rather large already.

Indeed, I'll try to get around to improving this.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Another update on the Guix Data Service

[Ludovic Courtès]

Hi!

I’m late to the party and learned about most of these things in the
meantime at the Guix Days, and this is all very cool!

Christopher Baines <mail@cbaines.net> skribis:

> There's now a page to compare two derivations [4], it's not a
> particularly clear comparison yet, but can hopefully be improved in the
> future. This is linked to from the derivation history pages [5].
>
> 4: http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv&target_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv
> 5: http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history

I didn’t know about this and it’s really cool!  It’s often necessary to
compare derivations when you want to understand what’s going on.

Do you think there’s derivation comparison code out there that could
make it into (guix derivations) or similar, with an eye on perhaps
having a CLI derivation comparison tool using the same code in the
future?

> Provide narinfo and nar files for derivations, and the required source
> files ([12] for example). This means the Guix Data Service can be used
> as a substitute server for derivations, for example [13].
>
> 12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo
> 13: guix build --substitute-urls="http://data.guix.gnu.org https://ci.guix.gnu.org" /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv

Well done.  :-)

I’m interested in making sure we can share the relevant code between
(guix scripts publish) and the Data Service if that’s not already the
case.

> In particular, I'm still thinking about;
>
>  - Automated code review for Guix patches, which I was working on when I
>    started working on the Guix Data Service, and most of the
>    functionality can be beneficial there.

Yeah, I think at the Guix Days you showed how to take advantage of the
Data Service while reviewing patches; perhaps we should write down a
tutorial or something.

>  - The Guix Weekly News project I was working on recently uses data from
>    the Guix Data Service, and that's what motivated storing the channel
>    news entries.

Yup, that looked nice!  What’s missing, actually?

>  - Trust in builds, once the data about package build reproducibility is
>    more complete, hopefully that will be informative and show where
>    build reproducibility issues can be fixed.
>
>  - Improving the speed in which substitutes are made available.
>
>  - Explaining the data that underpins Guix to a technical audience, like
>    derivations and how they work.

Nice!

Thanks,
Ludo’.

Re: Another update on the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2977 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> There's now a page to compare two derivations [4], it's not a
>> particularly clear comparison yet, but can hopefully be improved in the
>> future. This is linked to from the derivation history pages [5].
>>
>> 4: http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv&target_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv
>> 5: http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history
>
> I didn’t know about this and it’s really cool!  It’s often necessary to
> compare derivations when you want to understand what’s going on.
>
> Do you think there’s derivation comparison code out there that could
> make it into (guix derivations) or similar, with an eye on perhaps
> having a CLI derivation comparison tool using the same code in the
> future?

I'm not sure any of the Guix Data Service code is worth moving across,
but it might be worth trying to make something better that uses the
records in Guix, and then using that in the Guix Data Service.

>> Provide narinfo and nar files for derivations, and the required source
>> files ([12] for example). This means the Guix Data Service can be used
>> as a substitute server for derivations, for example [13].
>>
>> 12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo
>> 13: guix build --substitute-urls="http://data.guix.gnu.org https://ci.guix.gnu.org" /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv
>
> Well done.  :-)
>
> I’m interested in making sure we can share the relevant code between
> (guix scripts publish) and the Data Service if that’s not already the
> case.

I don't think I had any big problems duplicating code in (guix scripts
publish), however, it might be worth looking at perhaps trying to
identify similarities if there are cases where duplication can be
avoided.

>> In particular, I'm still thinking about;
>>
>>  - Automated code review for Guix patches, which I was working on when I
>>    started working on the Guix Data Service, and most of the
>>    functionality can be beneficial there.
>
> Yeah, I think at the Guix Days you showed how to take advantage of the
> Data Service while reviewing patches; perhaps we should write down a
> tutorial or something.

Yep, I'm definately hoping it's getting close to the point where it can
be useful.

>>  - The Guix Weekly News project I was working on recently uses data from
>>    the Guix Data Service, and that's what motivated storing the channel
>>    news entries.
>
> Yup, that looked nice!  What’s missing, actually?

I haven't sorted anything out to automatically update the data, and
"publish" the news articles.

I also need to improve the large lists of packages, as that isn't very
useful.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]