From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: More progress with the Guix Data Service Date: Fri, 17 May 2019 08:56:38 +0100 Message-ID: <87pnohms3t.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([209.51.188.92]:52533) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hRXjU-0000SW-AK for guix-devel@gnu.org; Fri, 17 May 2019 03:56:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hRXjS-0008Qr-VT for guix-devel@gnu.org; Fri, 17 May 2019 03:56:48 -0400 Received: from mira.cbaines.net ([212.71.252.8]:36316) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hRXjS-0008P3-Mf for guix-devel@gnu.org; Fri, 17 May 2019 03:56:46 -0400 Received: from localhost (cpc102582-walt20-2-0-cust14.13-2.cable.virginm.net [86.27.34.15]) by mira.cbaines.net (Postfix) with ESMTPSA id E582F16F5C for ; Fri, 17 May 2019 08:56:43 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id c7e4dc76 for ; Fri, 17 May 2019 07:56:43 +0000 (UTC) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --=-=-= Content-Type: text/plain Hey, In summary, I think the Guix Data Service might be getting useful enough that setting it up properly might be a good next step, and I'd be interested in what others think? A bit over a month ago, I sent out an update about one of the things I've been working on, something I've been calling the "Guix Data Service" [1]. 1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html Since then, I've made some more progress. There's a new statistics page [2]. I've got used to using Sqitch [3] to help manage the database schema, and I've added some basic tests. 2: https://prototype-guix-data-service.cbaines.net/statistic 3: http://guix.gnu.org/packages/sqitch-0.9999/ The error handling for loading new revisions is also more resilient now. As well as listening to the Guix Commits mailing list for emails about new revisions, more of the information in these emails is now stored, in particular, the time they were sent, and the branch the email applies to. This can be seen on the new Branches page [4]. 4: https://prototype-guix-data-service.cbaines.net/branches The content negotiation a little bit, at least in terms of the code, and JSON output support has been added to more pages. There's now a basic search function on the packages page [5], and the location, and the licenses for packages is now being stored (which can be seen on the page for a package, for example [6]). 5: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000 6: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha The location and license information is something I added specifically, as I noticed the Repology [7] service scraped [8] these from the Guix website. 7: https://repology.org/ 8: https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py While the Guix Data Service started as something to enable better understanding patches in an automated way, I think there are more uses for it, and initially, it's probably better to focus on the simple ones. The Repology use case is pretty simple, I think ideally there would be some machine readable data about the current state of packages in Guix available over the internet, and Repology would be able to download that on a regular basis. The URL is a bit long, but I think that is now close to being possible with the Guix data service. I haven't got something working yet to easily access data for the latest revision, but for a particular revision, you can request a JSON file containing all the information I think Repology currently gets about all packages. For example: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999 This is just the software side of the problem though. If this was to be used by Repology, it would have to be a more permanent thing, similar to the Cuirass and Mumi services that are currently setup around Guix. Does anyone have any thoughts on this? Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAlzeaTZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9Xc2KhAAt7bjAGo4ZOh6fxvPpwqpH0haFamwPfmVaN2EqD/CsuxoY02J9vvkRLN+ 6ZmIF3gezyZQJi1bBfpElr2yXPBSMBcWm12VKUiLS435E4DfITKS/WHHD3vHmzn5 rzuXTMUAIpASC+d0CVeTZO3HtLKqDAJGrMo+vZKYYg/iKNb7fRVp2rk7LpN8iQgz yCLxQl5PGtIt74BMWAohL8jvCYnI+xK56iN8So9Mg5IqQXIePgsuGO722HcdOSjQ Bm5YV9jt0Xx/jsg2hefjl7l88r+26DiV7BSPXz+b1i3crVgnhNHWLUDrnmCqck3Z pD3ysm0u+EUmZUKjyNXb4B3HR4a68WR61IkGL0XhrtF3p4yYw/g9IdS255H1Mb56 0SS5Oo5uzmQQJUi4TYVWY5BwzscSvy6mjgUsc9cLqYd9wXcq4s1CHx0gvUM5lZq6 d7d1W8CFn6UaHiMPLZvpyjeHt6O7nYdk9tqlmIxJbHKcLWzsrnPu1H8tWMNkzxy7 Mwyd8V4ws11rETwT2mt6kvxvkUuNCClOj/lffvwox2Td+R7bq3086DbpprW59Xiy Hie75DEkZgXfw8xLkQqoPXexRXOLyCGfNhebQVFTKIbM+vSmKPqbOcQhGfeihV+W ODhNgv8qatC9sSizCrzbnoOlUl6MEIxkxSX8Z8f7lSNNe7DLN/I= =V+ni -----END PGP SIGNATURE----- --=-=-=--