Re: Getting build information in to the Guix Data Service (draft patch)

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 3097 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> This is what I've currently tried to implement. The patch I'll send adds
>> two new tables to the Cuirass database, one to store events relating to
>> builds (like it being scheduled, or succeeding), and another to store
>> the ids of events which haven't yet been sent out. The code relating to
>> builds is then adjusted to populate these tables, and a new binary is
>> added to query for unsent events, and then send them out to some URL.
>
> So every event only has two states (sent/unsent), which means we assume
> there’s a single subscriber, right?  (Not a limitation because we could
> use a dedicated hub on top of that like you write, but I want to make
> sure I understand correctly.)

Yep, and that state is tracked through the BuildEventsOutbox table. An
entry in there means the event hasn't been sent out yet.

>> In the short term, the destination would be the Guix Data Service. In
>> the longer term, I think it would be better to send events to a WebSub
>> style hub, which then would distribute the events to one or more
>> subscribers.
>
> That sounds great.

Good good :)

...

>> +CREATE TABLE BuildEvents (
>> +  id          INTEGER PRIMARY KEY,
>> +  derivation  TEXT NOT NULL,
>> +  timestamp   INTEGER NOT NULL,
>> +  event       TEXT NOT NULL,
>> +  FOREIGN KEY (derivation) REFERENCES Builds (derivation)
>> +);
>
> This assumes build events are necessarily related to a derivation,
> though one could imagine events having to do with evaluations, jobsets,
> etc.
>
> Should ‘BuildEvents’ be more generic and have ‘event’ be an sexp or JSON
> string that could describe any kind of event?
>
> If we did that, we could keep ‘derivation’ but remove “NOT NULL” so that
> non-derivation events can exist but we can still query
> derivation-related events quickly.  Does that make sense?

Yep, that makes sense.

This seems to be the general decision about the way you use a relational
database, do you have specific tables (types) for the data, or do you
have a more freeform structure (columns containing sexp or JSON).

There's quite a few factors to consider here, the internals of Cuirass,
how these events are exposed through the HTTP API, how these events one
day might be published to a WebSub hub and then what kind of
subscriptions you might support in Cuirass (events for an individual
derivation, all builds for an evaluation, all builds, ...). I'll think
about it further and see if I can form an opinion either way.

>> +CREATE TABLE BuildEventsOutbox (
>> +  build_event_id INTEGER NOT NULL,
>> +  FOREIGN KEY (build_event_id) REFERENCES BuildEvents (id)
>> +);
>
> These are events that have not yet been sent, right?

Yep, exactly.

> Thanks!

Thanks for taking a look. I'll neaten up the patch a bit, add in some
error handling and retrying for sending out the events, and think a bit
more about the data model, then hopefully send an updated patch soon!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]