From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Progress with the Guix Data Service Date: Thu, 04 Apr 2019 23:59:43 +0100 Message-ID: <87a7h5e5r4.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([209.51.188.92]:34532) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hCBKw-0004bX-JT for guix-devel@gnu.org; Thu, 04 Apr 2019 18:59:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hCBKv-0006Bi-4r for guix-devel@gnu.org; Thu, 04 Apr 2019 18:59:58 -0400 Received: from li622-129.members.linode.com ([212.71.249.129]:56660 helo=mira.cbaines.net) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hCBKs-0005jQ-V2 for guix-devel@gnu.org; Thu, 04 Apr 2019 18:59:56 -0400 Received: from localhost (unknown [194.168.26.181]) by mira.cbaines.net (Postfix) with ESMTPSA id 47A1716D98 for ; Thu, 4 Apr 2019 23:59:47 +0100 (BST) Received: from phact (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id b1505c15 for ; Thu, 4 Apr 2019 22:59:46 +0000 (UTC) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --=-=-= Content-Type: text/plain So, in my last email featuring the Guix Data Service [1], I focused on reviewing patches, but since then, I've been focusing on fixing some issues with the Guix Data Service itself, adding some more functionality, and making some necessary code improvements. 1: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00010.html Previously, in the Guix Data Service database, each package had a single related derivation. This was a bit of an oversight, as it didn't account for different architectures. I've now fixed this, and I'm now generating derivations for all architectures, as well as generating cross-built derivations. I'm not quite sure if the cross-built derivations are a useful thing to have, but one interesting thing about computing them is that it's very easy to see what you can cross build, and what you can't. You can see some counts of derivations for a revision, for example [2]. 2: https://prototype-guix-data-service.cbaines.net/revision/54c9d7bb69847c89a7193347f858bb4e9476f7df Another new feature is following the guix-commits mailing list to find out about new revisions, and load them in. Following the mailing list still seems a little crude, but I'm not sure of any other way to get near real time notifications. I think I've also made quite a few interface improvements, so maybe try clicking around if you haven't done recently [3]! 3: https://prototype-guix-data-service.cbaines.net/ Along with these things, I also loaded some information from the berlin.guixsd.org build farm in to the Guix data service. I queried berlin for derivations, and stored some information about any builds that berlin had performed [4]. I was looking at this as I think it would be a useful addition to the comparison functionally, being able to compare the build status. I don't think doing the query at comparison time is going to be quick enough when comparing lots of packages, which is why the information is fetched and stored in the database. 4: https://prototype-guix-data-service.cbaines.net/builds I haven't got very far in terms of using the limited build information that I fetched, but I'm very interested in the applications of this data in the future. For example, if in the future, there are multiple build farms (which would be good along with reproducible builds to provide confidence in the binary substitutes), then gathering up all the build information in to one place could help with identifying where there are reproducibility problems, by comparing the results from different build farms. As a proof of concept, querying berlin worked OK, but I don't think the polling approach is scaleable. I think it would be good if there was the option to subscribe to updates, builds changing status in this case, and then get notifications when this happens. This would mean that polling would be unnecessary, and there would be less risk of missing messages. I'm used to custom ways of doing this, but if we were going to go down this route, it would be good I think if any solution wasn't too specific to the Guix Data Service, and was open to other consumers as well. One standardised approach is called WebSub [5]. I haven't tried it before, but from reading the specification, I think if Cuirass provided an Atom feed containing the events (when the builds change status), then somehow sent these to a hub (the service in WebSub responsible for managing subscriptions and delivering notifications), then the hub would notify subscribers by sending them the events which they haven't received yet (in the form of an Atom feed). 5: https://www.w3.org/TR/websub/ I'm still a bit uncertain about whether WebSub is entirely appropriate. The involvement of Atom feeds seems more related to traditional publishing of written works, not machine readable messages, but architecturally, it seems OK, and I don't think there's anything wrong with implementing something inspired by the specification, but not fully adhering to it. I'm still really interested in everyone's thoughts about this, especially on how to get information about builds in to the Guix data service. For the moment though, I've gone back to trying to neaten up the Patchwork service, and write a Getmail service, as that's currently used for both Patchwork, and the Guix Data Service. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAlymjF9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XdQFA/+P8A1U64NSmtWrhP98BT0Vl05y3nLjCAk2juY7defqmROpSusBogtTBSx iXUJ81BHE+JzN+dxY5pJqik5Av3n6g7oqGNCz8ryXGz9QUVLJEyFNfqSIDlUzVUo Mu30PNPMwYv7xWsSRmCuZocwA/iLgmqFHPLQ2uOiPp6beaxwgIn9bzKiIfg6QoP/ Q9DOfRpq+0GzNr5aWQIomZdfVYGby/Xc/5quyjU20DyaKrUB+83upU0NqvKc+Oq6 u/T5Rxod5nIdbxrinlc9vOJnQPCjpOgttx5AluXLAhJfLx0zTs+VSmDfK/y+QXxc UQfaEJ2V/8gRyMZU1qtfuTIwc4RUpe1hl5+8MhEQztBMXnA6245idJV9Mx/RfYy1 zU4ziwf0ARL+estv2HPSn10GijQnt7UBSwVDY0EeSgpIyjlvGGQU4raOqm1Y7+Xk vzwYwgpBx2qPNsLhb6MSv2N8e3mgmawUV2w9BNkyZ1OOnvFpmLwfQz1vRxWPlR7M Zfe6L9EE4mSXi7CuHo9eetp312ZWtTmaZFecJmUzmUCWtYeGL4h2flDNrgAsLTyp 1FjUTLnSSZ1RDQlCWsZuTNWbX6QG/FozL2sBYgfc0ZKCjJedf/XUJMs8PzeDktES jDLKTPaePYDrW1E0nBYUeyjZ34E2KGOAd3aTdkLFeyfyHZnDGRc= =cONX -----END PGP SIGNATURE----- --=-=-=--