February update on data.guix.gnu.org and the Guix Data Service

February update on data.guix.gnu.org and the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2978 bytes --]

Hey,

Another update on the Guix Data Service, I sent out the last update on
the 5th of January [1].

1: https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00073.html

Derivations for system tests [3], as well as channel instances [4]
(which relate to guix pull) are now captured. This is still a work in
progress, I think only the x86_64-linux derivations for the system tests
are captured, and the systems for the channel instances are limited by
what's available in the qemu-binfmt service.

3: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/system-tests
4: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/channel-instances

The way cross-built derivations are handled has changed. Previously the
system values were used, but I've now tried to move in the direction of
using GNU triplets. This can be seen on the revision pages in the
derivations table [5].

5: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c

It's now possible to configure the Guix Data Service to only process
certain branches by listing branches to explicitly include or
exclude. Using this feature, I've configured data.guix.gnu.org to only
look at master as this is what I want this instance of the Guix Data
Service to focus on. Previously, the jobs for core-updates, staging and
other branches could block processing revisions on the master branch, as
often they would take longer to process due to having to build lots of
packages just to build Guix. This issue would have only been amplified
now that data.guix.gnu.org is emulating other architectures and building
Guix for those as well.

Related to this, I've also added some code to enable removing data for a
branch, and removing unreferenced derivations. This is both for cleaning
up the data.guix.gnu.org database now I only want data for master, but
it should also be useful when using the Guix Data Service in an
environment where branches come and go, or be the basis of setting a
retention period for the data.

I forget exactly when, but recently I've been trying to revive the patch
review setup I was working on around a year ago [6]. I've setup an
instance of the Guix Data Service for this [7] (separate to the
data.guix.gnu.org one). I might try and have that instance of the Guix
Data Service process all the branches in the Guix git repository, now
that data.guix.gnu.org doesn't do that.

6: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html
7: https://guix-patches-data.cbaines.net/

Back to features though, the output from inferior processes used when
loading data for a revision is now captured and stored in the
database. This means you can see more of what's going on, like the
building of libgit2 here for example [8]. Previously you got less
information [9].

8: http://data.guix.gnu.org/job/14657
9: http://data.guix.gnu.org/job/14610

As always, let me know if you have any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: February update on data.guix.gnu.org and the Guix Data Service

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> I forget exactly when, but recently I've been trying to revive the patch
> review setup I was working on around a year ago [6]. I've setup an
> instance of the Guix Data Service for this [7] (separate to the
> data.guix.gnu.org one). I might try and have that instance of the Guix
> Data Service process all the branches in the Guix git repository, now
> that data.guix.gnu.org doesn't do that.
>
> 6: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html
> 7: https://guix-patches-data.cbaines.net/

This is great!
So how do we use it?
I tried clicking on series-2472, then "Latest processed revision", then
"View cgit".
Is it how it's supposed to be done?

I'm guessing the "series-N" naming is temporary, as it would be nice to
have commit / email title instead.  What does the number refer to?


Side question: I'd like to know if some package substitute is available,
say MAME:

http://data.guix.gnu.org/repository/1/branch/master/package/mame

If go to

http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/package/mame/0.218

I see it is scheduled.  If I click on "Scheduled" I end up here:

http://data.guix.gnu.org/build-server/1/build?derivation_file_name=/gnu/store/7aqmbfjwm0hiy1kviijj7fc9wqp71mnw-mame-0.218.drv

There is a timestamp: 2020-02-05T21:59:55
Does this mean it's been 13 days since the schedule?  That's odd.

Back to the
http://data.guix.gnu.org/repository/1/branch/master/package/mame page, I
don't really understand what the first version means.  If I click on the
latest date, I end up here:

http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c

This is Guix, not MAME, right?  I wonder why we display this here, I
find it a bit confusing.  I think the version table should explicitly
mention that this is the Guix revision, not that of the package
(assuming I understood correctly).

Anyways, as usual thanks for the fantastic job!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: February update on data.guix.gnu.org and the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3544 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> I forget exactly when, but recently I've been trying to revive the patch
>> review setup I was working on around a year ago [6]. I've setup an
>> instance of the Guix Data Service for this [7] (separate to the
>> data.guix.gnu.org one). I might try and have that instance of the Guix
>> Data Service process all the branches in the Guix git repository, now
>> that data.guix.gnu.org doesn't do that.
>>
>> 6: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html
>> 7: https://guix-patches-data.cbaines.net/
>
> This is great!
> So how do we use it?
> I tried clicking on series-2472, then "Latest processed revision", then
> "View cgit".
> Is it how it's supposed to be done?
>
> I'm guessing the "series-N" naming is temporary, as it would be nice to
> have commit / email title instead.  What does the number refer to?

I've been looking at Patchwork for pulling together patch series, it
should link to the Guix Data Service comparison. For example, this is a
series [1]. The N in series-N is the Patchwork series number.

1: https://patchwork.cbaines.net/project/guix-patches/list/?series=3022

The individual patches in the series have checks, these are used to link
off to Laminar for the job, cgit for the Git branch, and the Guix Data
Service for the comparison, [2] for example.

2: https://patchwork.cbaines.net/patch/20433/

> Side question: I'd like to know if some package substitute is available,
> say MAME:
>
> http://data.guix.gnu.org/repository/1/branch/master/package/mame
>
> If go to
>
> http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/package/mame/0.218
>
> I see it is scheduled.  If I click on "Scheduled" I end up here:
>
> http://data.guix.gnu.org/build-server/1/build?derivation_file_name=/gnu/store/7aqmbfjwm0hiy1kviijj7fc9wqp71mnw-mame-0.218.drv
>
> There is a timestamp: 2020-02-05T21:59:55
> Does this mean it's been 13 days since the schedule?  That's odd.

The data about builds is still incomplete. I'm still working on querying
Cuirass to find out about builds, as well as getting Cuirass to push
data to the Guix Data Service so that the information on builds in the
Guix Data Service is up to date.

> Back to the
> http://data.guix.gnu.org/repository/1/branch/master/package/mame page, I
> don't really understand what the first version means.  If I click on the
> latest date, I end up here:
>
> http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c
>
> This is Guix, not MAME, right?  I wonder why we display this here, I
> find it a bit confusing.  I think the version table should explicitly
> mention that this is the Guix revision, not that of the package
> (assuming I understood correctly).

I'm not quite sure what you're referring to on that page when you say
"first version" or "latest date", but what this page is meant to show is
how the version of the package on the master branch changes over
time.

Each row of the table represents a time period, the From column links to
the Guix revision at the start of the period, and the "To" column links
to the Guix revision at the end of that period. The Version column gives
the package version for that period.

I realise the links in the the From and To columns aren't
intuitive. You're saying there's the possibility of confusing the Guix
revision and MAME releases here, right?

> Anyways, as usual thanks for the fantastic job!

You're welcome, thanks for taking a look :)

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: February update on data.guix.gnu.org and the Guix Data Service

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2275 bytes --]

Hi Christopher!

Thanks for the details!

>> Back to the
>> http://data.guix.gnu.org/repository/1/branch/master/package/mame page, I
>> don't really understand what the first version means.  If I click on the
>> latest date, I end up here:
>>
>> http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c
>>
>> This is Guix, not MAME, right?  I wonder why we display this here, I
>> find it a bit confusing.  I think the version table should explicitly
>> mention that this is the Guix revision, not that of the package
>> (assuming I understood correctly).
>
> I'm not quite sure what you're referring to on that page when you say
> "first version" or "latest date", but what this page is meant to show is
> how the version of the package on the master branch changes over
> time.
>
> Each row of the table represents a time period, the From column links to
> the Guix revision at the start of the period, and the "To" column links
> to the Guix revision at the end of that period. The Version column gives
> the package version for that period.
>
> I realise the links in the the From and To columns aren't
> intuitive. You're saying there's the possibility of confusing the Guix
> revision and MAME releases here, right?

Exactly.

I looked at it again and this time I think I understand your intent.

A quick fix would be to rename the links:

- Dates: "Guix 2020-02-02 11:33:01"
- Package Info: "Mame derivation" (instead of "More information").

That said, I wonder if the presentation is right.

- The table format is a bit counter intuitive for what really is a
  timeline, I think.
  It took me a while to see those gray rectangles ;)

- We should the first and the last derivation of a version, but we don't
  show the other ones in between.

- Why have a "To" column since the date should be that just before the
  next "From"?  It's a bit redundant.

What about this instead: Display a linear timeline of all versions, and
clicking on a version shows a new page with a timeline of all
derivations for that version.

Maybe better than a new page, we could stay on the same page and have an
expandable frame or something.

What do you think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: February update on data.guix.gnu.org and the Guix Data Service

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> Another update on the Guix Data Service, I sent out the last update on
> the 5th of January [1].
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00073.html

I hadn’t yet got around to reading this message.  We really need
Guix Weekly News.  :-)

> Derivations for system tests [3], as well as channel instances [4]
> (which relate to guix pull) are now captured. This is still a work in
> progress, I think only the x86_64-linux derivations for the system tests
> are captured, and the systems for the channel instances are limited by
> what's available in the qemu-binfmt service.
>
> 3: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/system-tests
> 4: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/channel-instances

Neat!

> The way cross-built derivations are handled has changed. Previously the
> system values were used, but I've now tried to move in the direction of
> using GNU triplets. This can be seen on the revision pages in the
> derivations table [5].
>
> 5: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c

Woohoo!

[...]

> Related to this, I've also added some code to enable removing data for a
> branch, and removing unreferenced derivations. This is both for cleaning
> up the data.guix.gnu.org database now I only want data for master, but
> it should also be useful when using the Guix Data Service in an
> environment where branches come and go, or be the basis of setting a
> retention period for the data.

Nice.  Were you concerned specifically of unbounded growth on the
data.guix.gnu.org instance?

It would be interesting for data.guix.gnu.org to keep as much data as
possible, including perhaps for branches that have been deleted.  Then
the problem becomes more of a UI problem: how to display, for instance,
only the “active” branches on the main page.

WDYT?

> I forget exactly when, but recently I've been trying to revive the patch
> review setup I was working on around a year ago [6]. I've setup an
> instance of the Guix Data Service for this [7] (separate to the
> data.guix.gnu.org one). I might try and have that instance of the Guix
> Data Service process all the branches in the Guix git repository, now
> that data.guix.gnu.org doesn't do that.
>
> 6: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html
> 7: https://guix-patches-data.cbaines.net/

Yay!

At the Guix Days, you also shown very concretely how people could take
advantage of data.guix.gnu.org in their patch review workflow.  Perhaps
it’d be useful to share that info here or in the manual, even if it’s
still evolving.

> Back to features though, the output from inferior processes used when
> loading data for a revision is now captured and stored in the
> database. This means you can see more of what's going on, like the
> building of libgit2 here for example [8]. Previously you got less
> information [9].
>
> 8: http://data.guix.gnu.org/job/14657
> 9: http://data.guix.gnu.org/job/14610

Great.

Could you make a support request on Savannah to enable commit
notifications on the repo?  That’d be awesome.  :-)

Thanks for the great update, as always!

Ludo’.

Re: February update on data.guix.gnu.org and the Guix Data Service

[Bengt Richter]

Hi,

On +2020-03-12 14:21:51 +0100, Ludovic Courtès wrote:
> Hi!
> 
> Christopher Baines <mail@cbaines.net> skribis:
> 
> > Another update on the Guix Data Service, I sent out the last update on
> > the 5th of January [1].
> >
> > 1: https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00073.html
> 
> I hadn’t yet got around to reading this message.  We really need
> Guix Weekly News.  :-)
>
+1 ;-)

Quick comments:
    1.  Looks really nice :)
    1a. I notice that inputs in
http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/formatted
        seem to be sorted by leading hash. Might it be nice to have a button to sort starting past the hash?
    1b. Q: other than for making list diffs show equality reliably, is the input ordering significant in terms of
        dependencies or process scheduling?

2. Are the http [sans 's'] links necessary?

[...]
> 
> Thanks for the great update, as always!
> 
> Ludo’.
> 

-- 
Regards,
Bengt Richter

Re: February update on data.guix.gnu.org and the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]


Bengt Richter <bokr@bokr.com> writes:

> Hi,
>
> On +2020-03-12 14:21:51 +0100, Ludovic Courtès wrote:
>> Hi!
>>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>> > Another update on the Guix Data Service, I sent out the last update on
>> > the 5th of January [1].
>> >
>> > 1: https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00073.html
>>
>> I hadn’t yet got around to reading this message.  We really need
>> Guix Weekly News.  :-)
>>
> +1 ;-)
>
> Quick comments:
>     1.  Looks really nice :)

Thanks :)

>     1a. I notice that inputs in
> http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/formatted
>         seem to be sorted by leading hash. Might it be nice to have a button to sort starting past the hash?

So, the intent with that page is that it's a more readable version of
the plain derivation representation:

  http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/plain

I do however, think that maybe the sorting should be different on the
Detail view for derivations.

>     1b. Q: other than for making list diffs show equality reliably, is the input ordering significant in terms of
>         dependencies or process scheduling?

So, I think that making the order deterministic is definitely important,
at least in the Guix Data Service for providing substitutes.

> 2. Are the http [sans 's'] links necessary?

Do you mean why is HTTP being used and not HTTPS, or are you talking
about the links on the page themselves?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: February update on data.guix.gnu.org and the Guix Data Service

[Bengt Richter]

Hi Chris,

On +2020-03-12 19:58:54 +0000, Christopher Baines wrote:
...
> > 2. Are the http [sans 's'] links necessary?
> 
> Do you mean why is HTTP being used and not HTTPS, or are you talking
> about the links on the page themselves?

Sorry, I meant "why is HTTP being used and not HTTPS".

I've gotten the impression that it's a compromise
having to do with making load balancing easier somehow,
but that's my ignorant speculation, which any real factoid
could improve on :)

-- 
Regards,
Bengt Richter

Re: February update on data.guix.gnu.org and the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]


Bengt Richter <bokr@bokr.com> writes:

> Hi Chris,
>
> On +2020-03-12 19:58:54 +0000, Christopher Baines wrote:
> ...
>> > 2. Are the http [sans 's'] links necessary?
>>
>> Do you mean why is HTTP being used and not HTTPS, or are you talking
>> about the links on the page themselves?
>
> Sorry, I meant "why is HTTP being used and not HTTPS".
>
> I've gotten the impression that it's a compromise
> having to do with making load balancing easier somehow,
> but that's my ignorant speculation, which any real factoid
> could improve on :)

Just because I haven't configured LetsEncrypt on the server yet. I
wasn't sure what details to use when configuring it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: February update on data.guix.gnu.org and the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 4205 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Derivations for system tests [3], as well as channel instances [4]
>> (which relate to guix pull) are now captured. This is still a work in
>> progress, I think only the x86_64-linux derivations for the system tests
>> are captured, and the systems for the channel instances are limited by
>> what's available in the qemu-binfmt service.
>>
>> 3: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/system-tests
>> 4: http://data.guix.gnu.org/revision/3dd311e3a059131ef245417106d4fb659222ef3c/channel-instances
>
> Neat!

One small thing to add here is that I've noticed the system test
derivations recorded in the Guix Data Service differ from what Cuirass
is building. I think this is because they're tweaked a bit in
system-test-job (guix ci):

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/ci.scm#n280

>> Related to this, I've also added some code to enable removing data for a
>> branch, and removing unreferenced derivations. This is both for cleaning
>> up the data.guix.gnu.org database now I only want data for master, but
>> it should also be useful when using the Guix Data Service in an
>> environment where branches come and go, or be the basis of setting a
>> retention period for the data.
>
> Nice.  Were you concerned specifically of unbounded growth on the
> data.guix.gnu.org instance?

Partially, although I was more trying to reduce the time between the
master branch changing and the information being available in the
guix-data-service.

Even when the channel-instance derivation was being computed just to
x86-64_linux, it could take up to ~24 hours for some core-updates
changes. Now that the channel-instance derivations are computed for
multiple architectures, I think having those jobs take a long time would
have impacted how fast the jobs for the master branch were processed.

> It would be interesting for data.guix.gnu.org to keep as much data as
> possible, including perhaps for branches that have been deleted.  Then
> the problem becomes more of a UI problem: how to display, for instance,
> only the “active” branches on the main page.
>
> WDYT?

So, after I sent this email, I setup the guix-patches-data.cbaines.net
instance of the Guix Data Service to process data for the main Guix git
repository, but for all branches.

The index page for the Guix Data Service doesn't list deleted branches,
but they are shown on the page for each repository, for example:

  https://guix-patches-data.cbaines.net/repository/2/

>> I forget exactly when, but recently I've been trying to revive the patch
>> review setup I was working on around a year ago [6]. I've setup an
>> instance of the Guix Data Service for this [7] (separate to the
>> data.guix.gnu.org one). I might try and have that instance of the Guix
>> Data Service process all the branches in the Guix git repository, now
>> that data.guix.gnu.org doesn't do that.
>>
>> 6: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00089.html
>> 7: https://guix-patches-data.cbaines.net/
>
> Yay!
>
> At the Guix Days, you also shown very concretely how people could take
> advantage of data.guix.gnu.org in their patch review workflow.  Perhaps
> it’d be useful to share that info here or in the manual, even if it’s
> still evolving.

Indeed, hopefully I can get around to looking at it again soon :)

>> Back to features though, the output from inferior processes used when
>> loading data for a revision is now captured and stored in the
>> database. This means you can see more of what's going on, like the
>> building of libgit2 here for example [8]. Previously you got less
>> information [9].
>>
>> 8: http://data.guix.gnu.org/job/14657
>> 9: http://data.guix.gnu.org/job/14610
>
> Great.
>
> Could you make a support request on Savannah to enable commit
> notifications on the repo?  That’d be awesome.  :-)

Sure, I've created a ticket now:
https://savannah.nongnu.org/support/index.php?110208

> Thanks for the great update, as always!

You're welcome :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]