From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id Ana8MytxtF7vbgAA0tVLHw (envelope-from ) for ; Thu, 07 May 2020 20:35:55 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id CI+ZCzhxtF4cDgAA1q6Kng (envelope-from ) for ; Thu, 07 May 2020 20:36:08 +0000 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:470:142::17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 06C20940C62 for ; Thu, 7 May 2020 20:36:06 +0000 (UTC) Received: from localhost ([::1]:54374 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWnFW-00085j-JP for larch@yhetil.org; Thu, 07 May 2020 16:36:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46570) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWnFN-00084q-Rk for guix-devel@gnu.org; Thu, 07 May 2020 16:35:57 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:41419) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWnFM-0005sb-Cp for guix-devel@gnu.org; Thu, 07 May 2020 16:35:57 -0400 Received: from localhost (unknown [46.237.174.28]) by mira.cbaines.net (Postfix) with ESMTPSA id CCFDA27BBE1 for ; Thu, 7 May 2020 21:35:53 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 84ffd990 for ; Thu, 7 May 2020 20:35:51 +0000 (UTC) User-agent: mu4e 1.2.0; emacs 26.3 From: Christopher Baines To: guix-devel@gnu.org Subject: April update on data.guix.gnu.org (Guix Data Service) Date: Thu, 07 May 2020 21:35:48 +0100 Message-ID: <87v9l7fpp7.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/07 16:35:54 X-ACL-Warn: Detected OS = ??? X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 X-Spam-Score: -0.61 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Scan-Result: default: False [-0.61 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.49825473070812]; DWL_DNSWL_BLOCKED(0.00)[2001:470:142::17:from]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:142::/48:c]; R_MISSING_CHARSET(2.50)[]; TO_DN_NONE(0.00)[]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.10), country: US(-0.00), ip: 2001:470:142::17(-0.50)]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; MAILLIST(-0.20)[mailman]; SIGNED_PGP(-2.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:22989, ipnet:2001:470:142::/48, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[larch=yhetil.org]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; FROM_NEQ_ENVFROM(0.00)[mail@cbaines.net,guix-devel-bounces@gnu.org]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[cbaines.net:url,gnu.org:url,outreachy.org:url]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[guix-devel@gnu.org]; DMARC_NA(0.00)[cbaines.net]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: fK25T/oLN1ro --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi, This follows on from the email I send at the end of March [1]. 1: https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00454.html Some notable and exciting things have happened in April, so I wanted to send out another update. Firstly, there's now commits from 3 people in the Git repository, up from 2 last month. Thanks Vincent for sending in a patch :) Probably the most significant thing that happened recently is that Danjela was accepted for the Outreach internship related to the Guix Data Service and internationalization support [2]. Outreachy internships last ~3 months, and for the May 2020 period they formally start on the 19th of May (so a week and a bit from the sending of this email). Danjela has already made some good improvements to the Guix Data Service, so I'm sure there's more to come over the next few months. 2: https://www.outreachy.org/alums/ Some of the things Danjela worked on over the last month include making it possible to filter by locale on the lint warnings page [3], adding a JSON representation for the jobs page [4], and a plain text representation of the log for a job [5]. 3: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2= f7e/lint-warnings 4: https://data.guix.gnu.org/jobs.json?limit_results=3D20 5: https://data.guix.gnu.org/job/15897.txt One small change I made in relation to the excellent work happening around the GNU Hurd is to add i586-gnu as a system you can filter by. This will come in useful once core-updates is merged. In relation to the work I've been doing on the new Guix Build Coordinator [6], I did some work on the package derivations page for a revision. You can now avoid getting the data about builds which makes the page load a lot faster [7], and get the data in JSON. There's a script [8] in the guix-build-coordinator Git repository that can query this page and enqueue builds for the derivations. 6: https://git.cbaines.net/guix/build-coordinator/about/ 7: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2= f7e/package-derivations?search_query=3D&system=3Dx86_64-linux&target=3Dnone= &minimum_builds=3D&maximum_builds=3D&field=3D%28no-additional-fields%29&aft= er_name=3D&limit_results=3D&all_results=3Don 8: https://git.cbaines.net/guix/build-coordinator/tree/scripts/guix-build-c= oordinator-queue-builds-from-guix-data-service.in Thinking about substitute availability, I added some filters to the not so visible package derivation outputs page [9]. These allow you to filter for substitute availability, including or excluding outputs based on whether they're available or not from different substitute servers. 9: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab2= f7e/package-derivation-outputs This page is linked to from a new page I worked on [10], which shows substitute availability in a similar style to the package reproducibility page. This is still very much a work in progress, so don't take the data on that page too seriously. It's probably more a representation of how up to date the information the Guix Data Service has regarding substitutes is. 10: https://data.guix.gnu.org/revision/8ba2aa22f1b972b0bb0844c6ad1557b44eab= 2f7e/package-substitute-availability I worked a bit on some strange packages, mostly because the data was causing issues with the package reproducibility page. dev86 is a good example of this strange behaviour, if you ask Guix for a derivation for dev86, it doesn't matter what system you ask for, you'll always get a derivation for i686-linux. The Guix Data Service now double checks that when it asked for a derivation for a particular system, that was actually what was generated. If you search the logs (like [11]) for recent revisions for "produced a derivation for system", you'll see which derivations are not being ignored. As well as fixing this going forward, I also removed the strange data from the database. 11: https://data.guix.gnu.org/job/15897 There were some strange errors relating to handling of the log data, which I added some guards against. I also fixed up some issues processing old Guix revisions. There's still ~600 revisions [12] queued up for processing from early 2019. 12: https://data.guix.gnu.org/jobs/queue In terms of operations for data.guix.gnu.org, I wanted to track the disk space usage, so I did some work on Prometheus [13] and I also setup Grafana [14]. This has already proved useful in terms of spotting potential problems with disk space usage before things break, and I hope to setup some alerting at some point so that I don't even have to look at the data regularly. The setup seems to be reasonably stable at the moment, but I am going to need to do some work to allow the database to grow further some point within the next month or two. My plan currently is to add an additional volume to the virtual machine, and using the tablespaces feature of PostgreSQL, move a few of the larger tables over. 13: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D40738 14: http://mago.cbaines.net:3000/d/rYdddlPWk/node-exporter-full?orgId=3D1&v= ar-DS_PROMETHEUS=3DPrometheus&var-job=3Ddata.guix.gnu.org%2Fnode-exporter&v= ar-name=3Dguix-hetzner-1&var-node=3Ddata.guix.gnu.org&var-port=3D9100&from= =3Dnow-14d&to=3Dnow As always, if you have any questions or comments, just let me know :) Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl60cSRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XcTmBAArsPv3cJAbu57YGwQGGhK/lGQJpolXjCQZO47AzfMb80mfzNQu+N1jMAS tNxL0RKjFK2625mft7c9pbUbT4Z2SYygCADwdbSkavciHQugdeqMrRW4Ptb6WDhy zebMODhttZy/USlWgcGRNVNkUTE0BdAdAGVq50jOyWqNGR96Yb9rWypD5RmisVjt pWVl9KBGjcKx0s2NaAkD5LUaGDZ3F1oO+8jkhKKerKGlIRFNmjMAdwKxsudB/FHY sHi7hBqLC96S05ubBPG9GClxcqiRCsooUDnT07Vw1/YAVIudKHZ/NxPv2FPEUaXl 0x3csZJ/w7oSPPoSjZf3Nw3MykSXR25htpKpeh2aVU/Oi8zxki1gcenSY7ohDcxx E1rE9qRGMs9N2zjiW8mc838SSudJY1WYjo86uf6BsRY4oXNs+zSEJvsZGHSmwlRL gU8XiACO6JkX1gl+3i9WEJ9GT5ISQBgGU+djiyHFTgW1WooIKH5nr77jwTdJehtk vbf6wuUsG2uFlGdqHQeiifXv60rW4omPIMO2qoDevH0gM0S5EwjsP7htXvP7hfw7 5OrxoA4ljgG7aIJ/loOlI9k/MRfFHD9Xzq/do5c268gSxzy3KozGKBWL7Xmmzg+t CX+42oM4l+hd+txUxEF35XpLtE6uJDdKzJjBw5KG3bTVrmL0fW0= =rGrR -----END PGP SIGNATURE----- --=-=-=--