From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Another update on the Guix Data Service Date: Sun, 05 Jan 2020 20:15:37 +0000 Message-ID: <87v9pp64nq.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:53103) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ioCJQ-0006hr-ES for guix-devel@gnu.org; Sun, 05 Jan 2020 15:15:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ioCJO-0005B7-Q6 for guix-devel@gnu.org; Sun, 05 Jan 2020 15:15:48 -0500 Received: from mira.cbaines.net ([212.71.252.8]:36080) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ioCJO-00058V-HT for guix-devel@gnu.org; Sun, 05 Jan 2020 15:15:46 -0500 Received: from localhost (unknown [46.237.160.207]) by mira.cbaines.net (Postfix) with ESMTPSA id 4A700179F0 for ; Sun, 5 Jan 2020 20:15:45 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id d2ca6911 for ; Sun, 5 Jan 2020 20:15:42 +0000 (UTC) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --=-=-= Content-Type: text/plain Hey, It's been a while since I send out an update about the Guix Data Service, so here's a somewhat failed attempt at a short update about recent progress. There are formatted [1] and plain views of derivations [2]. The plain view is constructed from the representation in the database, but it should match what Guix produces. 1: http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/formatted 2: http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/plain Channel news entries are recorded in the database, you can see all that are present in a revision [3], but the comparison page should show new news entries as well. 3: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news There's now a page to compare two derivations [4], it's not a particularly clear comparison yet, but can hopefully be improved in the future. This is linked to from the derivation history pages [5]. 4: http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv&target_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv 5: http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history There's support for receiving build information from build servers, but this hasn't been used yet. I'm slowly looking at getting support for using this in to Cuirass [6] 6: https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00467.html Information about nars (substitutes) can now be stored ([7] for example). This data also shows up on a page for derivation outputs for a revision ([8] for example). 7: http://data.guix.gnu.org/gnu/store/001j0wk7srm7jlnd9lai7p1yzg4bdl3k-dunst-1.4.1 8: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/derivation-outputs There's a page for builds for a revision [9]. There's also a page for individual builds [10]. 9: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/builds 10: http://data.guix.gnu.org/build-server/2/build?derivation_file_name=/gnu/store/yc0pb7c0c198nb68c5wha41brzvrls01-k3b-19.08.3.drv The derivation outputs page for a revision shows information about outputs matching across build servers, but there's also a page about package reproducibility for a revision [11]. 11: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/package-reproducibility Provide narinfo and nar files for derivations, and the required source files ([12] for example). This means the Guix Data Service can be used as a substitute server for derivations, for example [13]. 12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo 13: guix build --substitute-urls="http://data.guix.gnu.org https://ci.guix.gnu.org" /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv Fix generating derivations for foreign architectures. There was an issue where only the x86_64-linux and i686-linux derivations were properly loaded, but now that's been fixed. I added support for loading mbox files a while ago, but I've only now got around to testing a few older revisions, and now I've loaded the mbox files for the first 3 months from 2019 so eventually those will be loaded in. A few things above were influenced by thinking about reproducible builds in the run up, during and after the reproducible builds summit at the start of December. I'm excited that there's some support in the Guix Data Service for comparing builds across build farms, but there's still a lot to do in terms of getting every derivation built by at least 2 separate build farms, and then getting that data promptly in to the Guix Data Service for comparison. Also at the reproducible builds summit, Ludovic made me aware that you can tell Guix to build a derivation, and if it doesn't have it locally, it will attempt to download it from substitute servers. I've now got the Guix Data Service setup to act as a substitute server for derivations, which while on it's own isn't particularly exciting (as build farms already can provide substitutes for derivations), it's now possible to get the information about derivations within the Guix Data Service, out of the Guix Data Service in a usable manor. This might have applications in the future, like speeding up guix pull by providing the derivation that corresponds to a revision, or providing derivations for a build farm to build. Like the example of providing substitutes for derivations, quite a lot of this work isn't directly impactful (at least yet), but I'm hoping it's laid the foundations for some more directly impactful stuff. In particular, I'm still thinking about; - Automated code review for Guix patches, which I was working on when I started working on the Guix Data Service, and most of the functionality can be beneficial there. - The Guix Weekly News project I was working on recently uses data from the Guix Data Service, and that's what motivated storing the channel news entries. - Trust in builds, once the data about package build reproducibility is more complete, hopefully that will be informative and show where build reproducibility issues can be fixed. - Improving the speed in which substitutes are made available. - Explaining the data that underpins Guix to a technical audience, like derivations and how they work. Do let me know if you have any thoughts or questions, I'm also going to be around at the Guix Days at the end of January. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl4SQ+lfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XeBJQ/+O3FZ8EPCQpX4GFZDodTKa12G/pPATehxuPcmfwFSL1TDoc6fN2+m2TZ7 8bW1yRAVqdK4psJrRT9SmCUnXwnBvCOE2jaMLX5zqg5Ta+0UIq8mcoZk4IkmrK9x TCMvcgX6HJF63qEmFynxiLHsVGt449Nuq7HKVSOAC6qH3lc6AF5iU1TphU8GCzgX P1ud9FV7ddh/f8urZ7S53pTJl6j7hizVy3i3BrLjuoukzJjn07pwiUFuK0UPVIDr kYr7bt6z5vfq2b1dRIzL0VQD/aDJU8NXlGYMzMdf3NcrsB/VoxznM8YGXepZONUX t3o285FZfIs7QYFLcpN4nCodrURBhm7+uRTKObb+fIbHsi+fjFO42BcihCjDTZVW az5JX6riVdmCZ0h4tBSl2FVjNsSFGj60/2qbAoMyx221KaDmFZbnP+n1z+GR8xdQ LbYNeUOHDUqj6fYd0a560dH0PS0Ec93dinwcXitNx1pXGal59s7gVy+TiwbqU9OJ fBYqunIlgvhybkbCipY9QEeNgavu1PtD4W+xKapsy/4UY+UAkIgcTnU6T6154xIU 3XQEmWP2HcBpi0z1Ec51l81gvN24S0TGeZ/Dx+02z01EZHWxckg6FQxwfPSJ5LDV X1pGCOwYs2D7du2SJaAaltsPW31XNIOv67WmHyYucMEwokWyw/c= =xO5E -----END PGP SIGNATURE----- --=-=-=--