April update on data.guix.gnu.org (Guix Data Service)

April update on data.guix.gnu.org (Guix Data Service)

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 5394 bytes --]

Hi,

This follows on from the email I send at the end of March [1].

1: https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00454.html

Some notable and exciting things have happened in April, so I wanted to
send out another update.

Firstly, there's now commits from 3 people in the Git repository, up
from 2 last month. Thanks Vincent for sending in a patch :)

Probably the most significant thing that happened recently is that
Danjela was accepted for the Outreach internship related to the Guix
Data Service and internationalization support [2]. Outreachy internships
last ~3 months, and for the May 2020 period they formally start on the
19th of May (so a week and a bit from the sending of this
email). Danjela has already made some good improvements to the Guix Data
Service, so I'm sure there's more to come over the next few months.

2: https://www.outreachy.org/alums/

Some of the things Danjela worked on over the last month include making
it possible to filter by locale on the lint warnings page [3], adding a
JSON representation for the jobs page [4], and a plain text
representation of the log for a job [5].

3: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2f7e/lint-warnings
4: https://data.guix.gnu.org/jobs.json?limit_results=20
5: https://data.guix.gnu.org/job/15897.txt

One small change I made in relation to the excellent work happening
around the GNU Hurd is to add i586-gnu as a system you can filter
by. This will come in useful once core-updates is merged.

In relation to the work I've been doing on the new Guix Build
Coordinator [6], I did some work on the package derivations page for a
revision. You can now avoid getting the data about builds which makes
the page load a lot faster [7], and get the data in JSON. There's a
script [8] in the guix-build-coordinator Git repository that can query
this page and enqueue builds for the derivations.

6: https://git.cbaines.net/guix/build-coordinator/about/
7: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2f7e/package-derivations?search_query=&system=x86_64-linux&target=none&minimum_builds=&maximum_builds=&field=%28no-additional-fields%29&after_name=&limit_results=&all_results=on
8: https://git.cbaines.net/guix/build-coordinator/tree/scripts/guix-build-coordinator-queue-builds-from-guix-data-service.in

Thinking about substitute availability, I added some filters to the not
so visible package derivation outputs page [9]. These allow you to
filter for substitute availability, including or excluding outputs based
on whether they're available or not from different substitute servers.

9: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2f7e/package-derivation-outputs

This page is linked to from a new page I worked on [10], which shows
substitute availability in a similar style to the package
reproducibility page. This is still very much a work in progress, so
don't take the data on that page too seriously. It's probably more a
representation of how up to date the information the Guix Data Service
has regarding substitutes is.

10: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2f7e/package-substitute-availability

I worked a bit on some strange packages, mostly because the data was
causing issues with the package reproducibility page. dev86 is a good
example of this strange behaviour, if you ask Guix for a derivation for
dev86, it doesn't matter what system you ask for, you'll always get a
derivation for i686-linux. The Guix Data Service now double checks that
when it asked for a derivation for a particular system, that was
actually what was generated. If you search the logs (like [11]) for
recent revisions for "produced a derivation for system", you'll see
which derivations are not being ignored. As well as fixing this going
forward, I also removed the strange data from the database.

11: https://data.guix.gnu.org/job/15897

There were some strange errors relating to handling of the log data,
which I added some guards against. I also fixed up some issues
processing old Guix revisions. There's still ~600 revisions [12] queued
up for processing from early 2019.

12: https://data.guix.gnu.org/jobs/queue

In terms of operations for data.guix.gnu.org, I wanted to track the disk
space usage, so I did some work on Prometheus [13] and I also setup
Grafana [14]. This has already proved useful in terms of spotting
potential problems with disk space usage before things break, and I hope
to setup some alerting at some point so that I don't even have to look
at the data regularly. The setup seems to be reasonably stable at the
moment, but I am going to need to do some work to allow the database to
grow further some point within the next month or two. My plan currently
is to add an additional volume to the virtual machine, and using the
tablespaces feature of PostgreSQL, move a few of the larger tables over.

13: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40738
14: http://mago.cbaines.net:3000/d/rYdddlPWk/node-exporter-full?orgId=1&var-DS_PROMETHEUS=Prometheus&var-job=data.guix.gnu.org%2Fnode-exporter&var-name=guix-hetzner-1&var-node=data.guix.gnu.org&var-port=9100&from=now-14d&to=now

As always, if you have any questions or comments, just let me know :)

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]