From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Re: Getting build information in to the Guix Data Service (draft patch) Date: Thu, 24 Oct 2019 00:32:06 +0100 Message-ID: <8736fjqbrd.fsf@cbaines.net> References: <87mudvrhim.fsf@cbaines.net> <87tv7zsezm.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:37463) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iNQ70-0003fe-OL for guix-devel@gnu.org; Wed, 23 Oct 2019 19:32:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iNQ6z-0003Nb-DT for guix-devel@gnu.org; Wed, 23 Oct 2019 19:32:18 -0400 In-reply-to: <87tv7zsezm.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> This is what I've currently tried to implement. The patch I'll send adds >> two new tables to the Cuirass database, one to store events relating to >> builds (like it being scheduled, or succeeding), and another to store >> the ids of events which haven't yet been sent out. The code relating to >> builds is then adjusted to populate these tables, and a new binary is >> added to query for unsent events, and then send them out to some URL. > > So every event only has two states (sent/unsent), which means we assume > there=E2=80=99s a single subscriber, right? (Not a limitation because we= could > use a dedicated hub on top of that like you write, but I want to make > sure I understand correctly.) Yep, and that state is tracked through the BuildEventsOutbox table. An entry in there means the event hasn't been sent out yet. >> In the short term, the destination would be the Guix Data Service. In >> the longer term, I think it would be better to send events to a WebSub >> style hub, which then would distribute the events to one or more >> subscribers. > > That sounds great. Good good :) ... >> +CREATE TABLE BuildEvents ( >> + id INTEGER PRIMARY KEY, >> + derivation TEXT NOT NULL, >> + timestamp INTEGER NOT NULL, >> + event TEXT NOT NULL, >> + FOREIGN KEY (derivation) REFERENCES Builds (derivation) >> +); > > This assumes build events are necessarily related to a derivation, > though one could imagine events having to do with evaluations, jobsets, > etc. > > Should =E2=80=98BuildEvents=E2=80=99 be more generic and have =E2=80=98ev= ent=E2=80=99 be an sexp or JSON > string that could describe any kind of event? > > If we did that, we could keep =E2=80=98derivation=E2=80=99 but remove =E2= =80=9CNOT NULL=E2=80=9D so that > non-derivation events can exist but we can still query > derivation-related events quickly. Does that make sense? Yep, that makes sense. This seems to be the general decision about the way you use a relational database, do you have specific tables (types) for the data, or do you have a more freeform structure (columns containing sexp or JSON). There's quite a few factors to consider here, the internals of Cuirass, how these events are exposed through the HTTP API, how these events one day might be published to a WebSub hub and then what kind of subscriptions you might support in Cuirass (events for an individual derivation, all builds for an evaluation, all builds, ...). I'll think about it further and see if I can form an opinion either way. >> +CREATE TABLE BuildEventsOutbox ( >> + build_event_id INTEGER NOT NULL, >> + FOREIGN KEY (build_event_id) REFERENCES BuildEvents (id) >> +); > > These are events that have not yet been sent, right? Yep, exactly. > Thanks! Thanks for taking a look. I'll neaten up the patch a bit, add in some error handling and retrying for sending out the events, and think a bit more about the data model, then hopefully send an updated patch soon! Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl2w4vZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XcCMRAAglzU0iilSqgvTmwJITxZz6qOPMjyUS4C7gO695XiKUCde6B0qy8dqTtO 20haxdxLc0OZKFh4Z5GPq17HUyN4jKR1LXqntUnVQzdaVbTmcluWnJM5vK4KLIud 4iwfjByGfB7g/ceWORq+PsrLerpy/qCPbmWwXk3UDfZYB5oHVCE+b3BExsHbZxO9 aHA9lA/6MKczylCAyf/ABZJZujBhFDiR8xu8bTPgMLVwUjV23hElDNBV03J/N0hu kQ48QW7KKw4+MbiLn9CF5S89pMKT+ak9O6NjgHoZvqV2Pc4E6j0vhcZc0k7JXM00 uC6hUBm/ERqHgzSw4NuyEXhPIKUYDDG8eUlJg7ME9jVpw5jfUZGHmfOUpirK0nhK wCMDuDgBElF6TJZj8S2e/f7XIQjbiXqcUq3F6b8Rv0QtSAaPXx97ORHfnEqRDZpU EMajcSSqCIStsZl1ibU2lQy8o2XQBzlwfWpyYQ1SxLZQMB3vZI4i5upxibMyTx/i 4/fKJSMOK6XnFmdXCf0oypMC65Il8OVAGQTpDwh0B0JQNTY3qXr/OCuBkUk/x4Qi ziDDmTCorbNCU/jv05ANV2cNcM3hQ1t75kQk26MUli2DA+4NwXuyQTt19J72uUh3 37PIVWroa/C/B+VCerFUEDvNYkPgw1Y0z0+iRf++0LohtRCH11s= =Ci0l -----END PGP SIGNATURE----- --=-=-=--