From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Re: More progress with the Guix Data Service Date: Fri, 17 May 2019 19:26:56 +0100 Message-ID: <87o941lyxb.fsf@cbaines.net> References: <87pnohms3t.fsf@cbaines.net> <9b0ce280d338340ea0621f46cb7d0377@hyper.dev> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([209.51.188.92]:55178) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hRhZS-0008TG-DZ for guix-devel@gnu.org; Fri, 17 May 2019 14:27:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hRhZQ-0001Ni-TI for guix-devel@gnu.org; Fri, 17 May 2019 14:27:06 -0400 In-reply-to: <9b0ce280d338340ea0621f46cb7d0377@hyper.dev> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Amirouche Cc: guix-devel@gnu.org, Guix-devel --=-=-= Content-Type: text/plain Amirouche writes: > On 2019-05-17 09:56, Christopher Baines wrote: >> Hey, >> >> In summary, I think the Guix Data Service might be getting useful >> enough >> that setting it up properly might be a good next step, and I'd be >> interested in what others think? > > Yeah. And it looks good :) Thanks :) >> A bit over a month ago, I sent out an update about one of the things >> I've been working on, something I've been calling the "Guix Data >> Service" [1]. >> >> 1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html >> >> Since then, I've made some more progress. There's a new statistics page >> [2]. I've got used to using Sqitch [3] to help manage the database >> schema, and I've added some basic tests. > > statistic page is a 404. I am very interested to learn how you use > sqitch. Sorry, the link is wrong, it's actually: https://prototype-guix-data-service.cbaines.net/statistics As for Sqitch, all I've been doing so far is sqitch add ... to create a change, sqitch deploy ... to apply changes to a database and sqitch revert ... to revert changes to a database. It's just some structure and tooling that I don't have to write :) >> 2: https://prototype-guix-data-service.cbaines.net/statistic >> 3: http://guix.gnu.org/packages/sqitch-0.9999/ >> >> The error handling for loading new revisions is also more resilient >> now. >> >> As well as listening to the Guix Commits mailing list for emails about >> new revisions, more of the information in these emails is now >> stored, in >> particular, the time they were sent, and the branch the email applies >> to. This can be seen on the new Branches page [4]. >> >> 4: https://prototype-guix-data-service.cbaines.net/branches > > Here when I click on a commit, I would expect the message of the commit > and the build status of the impacted softwares, e.g. > > https://prototype-guix-data-service.cbaines.net/revision/d59c90a5bfdd6b723bea939b8538c7c9b3c1b2a6 I've just realised the table headers on the branches page are the wrong way around. Anyway, I've added some information about git repositories to the database so I could link to cgit for the location of a package, so I'll have a look at also putting a link to commits on that page. As for build status information, this is something I talked a little more about in my last email (link at the top), but in summary, I've done some initial work, but more needs doing to get the build status information in to the database. >> The content negotiation a little bit, at least in terms of the code, >> and >> JSON output support has been added to more pages. >> >> There's now a basic search function on the packages page [5], and the >> location, and the licenses for packages is now being stored (which can >> be seen on the page for a package, for example [6]). >> >> 5: >> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000 >> 6: >> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha >> >> The location and license information is something I added specifically, >> as I noticed the Repology [7] service scraped [8] these from the Guix >> website. >> >> 7: https://repology.org/ >> 8: >> https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py >> >> While the Guix Data Service started as something to enable better >> understanding patches in an automated way, I think there are more uses >> for it, and initially, it's probably better to focus on the simple >> ones. > > It is not clear right now that it is related to patches, there is no > patch > anywhere to see. Indeed. What I've been using the Guix Data Service for is working out what a series of patches does. For example, on my test Patchwork instance, you can see this series of patches [1], and there's a corresponding page from the Guix Data Service showing what the effect of the patches is [2]. 1: https://patchwork.cbaines.net/project/guix-patches/list/?series=1048 2: https://prototype-guix-data-service.cbaines.net/compare?base_commit=4d46775b1d04f9edb6fd729cfcd4b745be74e03b&target_commit=b3cfde472e5b2e6a9489a1b6c437daa659a977a6 >> The Repology use case is pretty simple, I think ideally there would be >> some machine readable data about the current state of packages in Guix >> available over the internet, and Repology would be able to download >> that >> on a regular basis. > > Repology is nice, but I would prefer wikidata support. Interesting, I don't know much about wikidata, but taking Guile as an example, it looks like wikidata is tracking distributions that package Guile, along with the Free Software Directory [3] 3: https://www.wikidata.org/wiki/Q1486208#identifiers Linking to other data/representations of packages from the Guix Data Service has been something I've been thinking about. Especially things like the Free Software Directory. Maybe wikidata would be a good think to link to. I'd also be really interested what would be required to get wikidata linking to Guix. >> The URL is a bit long, but I think that is now close to being possible >> with the Guix data service. I haven't got something working yet to >> easily access data for the latest revision, but for a particular >> revision, you can request a JSON file containing all the information I >> think Repology currently gets about all packages. For example: >> >> >> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999 > > FWIW it is very slow. To be honest, I'm actually pretty happy with the speed! I was worried that NGinx was going to time out and that it wasn't even going to work. I think there's probably some optimisations I can do to the queries to make it faster, and I haven't added any caching anyware yet, and it should be possible to cache lots of things, which will help lots. >> Does anyone have any thoughts on this? > > It seems to me it would be useful to have that informations > somewhere. Like you > explain having the correct build information is a must have IMO along > the logs. > > Where is the code? In a git repository here: https://git.cbaines.net/guix/data-service/ Thanks for taking a look, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAlze/PBfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XceUg/8C3gOoRVDE8fmX/EcqWxhHKBYR6lGBvt7H+AjMs4Ci7vOxGs2phXsgnd3 rlJ3l+8hPJ47xrDelD6Hdh+XjCaDMVI8um9nZfRBfQOflA2rArWMb+LrtFXWRYI8 pWw3SSwEmGwGuCZ6ALey4e7xOHq2oForJ4VXF4+h3Fy4WZ/3AHotfuJAy36hZ8nb ZQm38IocWzXSqIRk9RzfFD5R9dw5d6Cpk7nPpXh/urtlE3iApooGvosaCIz2jBC2 HRObQDnj6aH1e5dr9IUmb/atC0LMtYsDCaHuQ/lNg2XSblKySWaeIMPynCTO2Iwl gJFTtEPvieCTuxQt7Q7PuaN4GoeH64pds2i5lmG8hFHVOEWDN0MUX9waL20VdCBq QKDnJ4OGnybl6tcab9O1/cWllkrXHOphK5AZMHaDNwQSZ4eM9afbgAXkeEdzuBwT gPUGC88j8wf45MB1CuYqAUOarpC2Fs9+tS1zRmKgDHVvLizE6HDKRs54Fn/zXJRD ZyLVUdUUihwivPJOC4sA75/ioH5QU2ucOoMlg0GiiCihcQmDj5Zyo5KtAN76CyIh 5VTJbDyU25IScF5gGUHq/HU2jcLzOBnpLyNDa/0lrpCybbdfP1sVu0lIhCASSgTX phjnNL+LuIhA0HjTuJ9p9LKtrf6qsYFXw9dda6nD0FR3JMaIOhk= =V8Eq -----END PGP SIGNATURE----- --=-=-=--