* More progress with the Guix Data Service
@ 2019-05-17 7:56 Christopher Baines
2019-05-17 10:07 ` Amirouche
2019-05-20 15:20 ` Ludovic Courtès
0 siblings, 2 replies; 7+ messages in thread
From: Christopher Baines @ 2019-05-17 7:56 UTC (permalink / raw)
To: guix-devel
[-- Attachment #1: Type: text/plain, Size: 3273 bytes --]
Hey,
In summary, I think the Guix Data Service might be getting useful enough
that setting it up properly might be a good next step, and I'd be
interested in what others think?
A bit over a month ago, I sent out an update about one of the things
I've been working on, something I've been calling the "Guix Data
Service" [1].
1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html
Since then, I've made some more progress. There's a new statistics page
[2]. I've got used to using Sqitch [3] to help manage the database
schema, and I've added some basic tests.
2: https://prototype-guix-data-service.cbaines.net/statistic
3: http://guix.gnu.org/packages/sqitch-0.9999/
The error handling for loading new revisions is also more resilient now.
As well as listening to the Guix Commits mailing list for emails about
new revisions, more of the information in these emails is now stored, in
particular, the time they were sent, and the branch the email applies
to. This can be seen on the new Branches page [4].
4: https://prototype-guix-data-service.cbaines.net/branches
The content negotiation a little bit, at least in terms of the code, and
JSON output support has been added to more pages.
There's now a basic search function on the packages page [5], and the
location, and the licenses for packages is now being stored (which can
be seen on the page for a package, for example [6]).
5: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
6: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha
The location and license information is something I added specifically,
as I noticed the Repology [7] service scraped [8] these from the Guix
website.
7: https://repology.org/
8: https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py
While the Guix Data Service started as something to enable better
understanding patches in an automated way, I think there are more uses
for it, and initially, it's probably better to focus on the simple ones.
The Repology use case is pretty simple, I think ideally there would be
some machine readable data about the current state of packages in Guix
available over the internet, and Repology would be able to download that
on a regular basis.
The URL is a bit long, but I think that is now close to being possible
with the Guix data service. I haven't got something working yet to
easily access data for the latest revision, but for a particular
revision, you can request a JSON file containing all the information I
think Repology currently gets about all packages. For example:
https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
This is just the software side of the problem though. If this was to be
used by Repology, it would have to be a more permanent thing, similar to
the Cuirass and Mumi services that are currently setup around Guix. Does
anyone have any thoughts on this?
Thanks,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: More progress with the Guix Data Service
2019-05-17 7:56 More progress with the Guix Data Service Christopher Baines
@ 2019-05-17 10:07 ` Amirouche
2019-05-17 18:26 ` Christopher Baines
2019-05-20 15:20 ` Ludovic Courtès
1 sibling, 1 reply; 7+ messages in thread
From: Amirouche @ 2019-05-17 10:07 UTC (permalink / raw)
To: Christopher Baines; +Cc: guix-devel, Guix-devel
On 2019-05-17 09:56, Christopher Baines wrote:
> Hey,
>
> In summary, I think the Guix Data Service might be getting useful
> enough
> that setting it up properly might be a good next step, and I'd be
> interested in what others think?
Yeah. And it looks good :)
> A bit over a month ago, I sent out an update about one of the things
> I've been working on, something I've been calling the "Guix Data
> Service" [1].
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html
>
> Since then, I've made some more progress. There's a new statistics page
> [2]. I've got used to using Sqitch [3] to help manage the database
> schema, and I've added some basic tests.
statistic page is a 404. I am very interested to learn how you use
sqitch.
>
> 2: https://prototype-guix-data-service.cbaines.net/statistic
> 3: http://guix.gnu.org/packages/sqitch-0.9999/
>
> The error handling for loading new revisions is also more resilient
> now.
>
> As well as listening to the Guix Commits mailing list for emails about
> new revisions, more of the information in these emails is now stored,
> in
> particular, the time they were sent, and the branch the email applies
> to. This can be seen on the new Branches page [4].
>
> 4: https://prototype-guix-data-service.cbaines.net/branches
Here when I click on a commit, I would expect the message of the commit
and the build status of the impacted softwares, e.g.
https://prototype-guix-data-service.cbaines.net/revision/d59c90a5bfdd6b723bea939b8538c7c9b3c1b2a6
>
> The content negotiation a little bit, at least in terms of the code,
> and
> JSON output support has been added to more pages.
>
> There's now a basic search function on the packages page [5], and the
> location, and the licenses for packages is now being stored (which can
> be seen on the page for a package, for example [6]).
>
> 5:
> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
> 6:
> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha
>
> The location and license information is something I added specifically,
> as I noticed the Repology [7] service scraped [8] these from the Guix
> website.
>
> 7: https://repology.org/
> 8:
> https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py
>
> While the Guix Data Service started as something to enable better
> understanding patches in an automated way, I think there are more uses
> for it, and initially, it's probably better to focus on the simple
> ones.
It is not clear right now that it is related to patches, there is no
patch
anywhere to see.
> The Repology use case is pretty simple, I think ideally there would be
> some machine readable data about the current state of packages in Guix
> available over the internet, and Repology would be able to download
> that
> on a regular basis.
Repology is nice, but I would prefer wikidata support.
>
> The URL is a bit long, but I think that is now close to being possible
> with the Guix data service. I haven't got something working yet to
> easily access data for the latest revision, but for a particular
> revision, you can request a JSON file containing all the information I
> think Repology currently gets about all packages. For example:
>
>
> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
FWIW it is very slow.
> Does anyone have any thoughts on this?
It seems to me it would be useful to have that informations somewhere.
Like you
explain having the correct build information is a must have IMO along
the logs.
Where is the code?
Thanks for sharing!
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: More progress with the Guix Data Service
2019-05-17 10:07 ` Amirouche
@ 2019-05-17 18:26 ` Christopher Baines
2019-05-20 15:23 ` Ludovic Courtès
0 siblings, 1 reply; 7+ messages in thread
From: Christopher Baines @ 2019-05-17 18:26 UTC (permalink / raw)
To: Amirouche; +Cc: guix-devel, Guix-devel
[-- Attachment #1: Type: text/plain, Size: 6478 bytes --]
Amirouche <amirouche@hyper.dev> writes:
> On 2019-05-17 09:56, Christopher Baines wrote:
>> Hey,
>>
>> In summary, I think the Guix Data Service might be getting useful
>> enough
>> that setting it up properly might be a good next step, and I'd be
>> interested in what others think?
>
> Yeah. And it looks good :)
Thanks :)
>> A bit over a month ago, I sent out an update about one of the things
>> I've been working on, something I've been calling the "Guix Data
>> Service" [1].
>>
>> 1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html
>>
>> Since then, I've made some more progress. There's a new statistics page
>> [2]. I've got used to using Sqitch [3] to help manage the database
>> schema, and I've added some basic tests.
>
> statistic page is a 404. I am very interested to learn how you use
> sqitch.
Sorry, the link is wrong, it's actually:
https://prototype-guix-data-service.cbaines.net/statistics
As for Sqitch, all I've been doing so far is sqitch add ... to create a
change, sqitch deploy ... to apply changes to a database and sqitch
revert ... to revert changes to a database. It's just some structure and
tooling that I don't have to write :)
>> 2: https://prototype-guix-data-service.cbaines.net/statistic
>> 3: http://guix.gnu.org/packages/sqitch-0.9999/
>>
>> The error handling for loading new revisions is also more resilient
>> now.
>>
>> As well as listening to the Guix Commits mailing list for emails about
>> new revisions, more of the information in these emails is now
>> stored, in
>> particular, the time they were sent, and the branch the email applies
>> to. This can be seen on the new Branches page [4].
>>
>> 4: https://prototype-guix-data-service.cbaines.net/branches
>
> Here when I click on a commit, I would expect the message of the commit
> and the build status of the impacted softwares, e.g.
>
> https://prototype-guix-data-service.cbaines.net/revision/d59c90a5bfdd6b723bea939b8538c7c9b3c1b2a6
I've just realised the table headers on the branches page are the wrong
way around.
Anyway, I've added some information about git repositories to the
database so I could link to cgit for the location of a package, so I'll
have a look at also putting a link to commits on that page.
As for build status information, this is something I talked a little
more about in my last email (link at the top), but in summary, I've done
some initial work, but more needs doing to get the build status
information in to the database.
>> The content negotiation a little bit, at least in terms of the code,
>> and
>> JSON output support has been added to more pages.
>>
>> There's now a basic search function on the packages page [5], and the
>> location, and the licenses for packages is now being stored (which can
>> be seen on the page for a package, for example [6]).
>>
>> 5:
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
>> 6:
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha
>>
>> The location and license information is something I added specifically,
>> as I noticed the Repology [7] service scraped [8] these from the Guix
>> website.
>>
>> 7: https://repology.org/
>> 8:
>> https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py
>>
>> While the Guix Data Service started as something to enable better
>> understanding patches in an automated way, I think there are more uses
>> for it, and initially, it's probably better to focus on the simple
>> ones.
>
> It is not clear right now that it is related to patches, there is no
> patch
> anywhere to see.
Indeed. What I've been using the Guix Data Service for is working out
what a series of patches does. For example, on my test Patchwork
instance, you can see this series of patches [1], and there's a
corresponding page from the Guix Data Service showing what the effect of
the patches is [2].
1: https://patchwork.cbaines.net/project/guix-patches/list/?series=1048
2: https://prototype-guix-data-service.cbaines.net/compare?base_commit=4d46775b1d04f9edb6fd729cfcd4b745be74e03b&target_commit=b3cfde472e5b2e6a9489a1b6c437daa659a977a6
>> The Repology use case is pretty simple, I think ideally there would be
>> some machine readable data about the current state of packages in Guix
>> available over the internet, and Repology would be able to download
>> that
>> on a regular basis.
>
> Repology is nice, but I would prefer wikidata support.
Interesting, I don't know much about wikidata, but taking Guile as an
example, it looks like wikidata is tracking distributions that package
Guile, along with the Free Software Directory [3]
3: https://www.wikidata.org/wiki/Q1486208#identifiers
Linking to other data/representations of packages from the Guix Data
Service has been something I've been thinking about. Especially things
like the Free Software Directory. Maybe wikidata would be a good think
to link to. I'd also be really interested what would be required to get
wikidata linking to Guix.
>> The URL is a bit long, but I think that is now close to being possible
>> with the Guix data service. I haven't got something working yet to
>> easily access data for the latest revision, but for a particular
>> revision, you can request a JSON file containing all the information I
>> think Repology currently gets about all packages. For example:
>>
>>
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
>
> FWIW it is very slow.
To be honest, I'm actually pretty happy with the speed! I was worried
that NGinx was going to time out and that it wasn't even going to work.
I think there's probably some optimisations I can do to the queries to
make it faster, and I haven't added any caching anyware yet, and it
should be possible to cache lots of things, which will help lots.
>> Does anyone have any thoughts on this?
>
> It seems to me it would be useful to have that informations
> somewhere. Like you
> explain having the correct build information is a must have IMO along
> the logs.
>
> Where is the code?
In a git repository here: https://git.cbaines.net/guix/data-service/
Thanks for taking a look,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: More progress with the Guix Data Service
2019-05-17 7:56 More progress with the Guix Data Service Christopher Baines
2019-05-17 10:07 ` Amirouche
@ 2019-05-20 15:20 ` Ludovic Courtès
2019-05-20 20:14 ` Christopher Baines
1 sibling, 1 reply; 7+ messages in thread
From: Ludovic Courtès @ 2019-05-20 15:20 UTC (permalink / raw)
To: Christopher Baines; +Cc: guix-devel
Hi!
Christopher Baines <mail@cbaines.net> skribis:
> Since then, I've made some more progress. There's a new statistics page
> [2]. I've got used to using Sqitch [3] to help manage the database
> schema, and I've added some basic tests.
The URL for [2] is actually:
https://prototype-guix-data-service.cbaines.net/statistics
(with an ‘s’.) See, I paid attention! ;-)
> As well as listening to the Guix Commits mailing list for emails about
> new revisions, more of the information in these emails is now stored, in
> particular, the time they were sent, and the branch the email applies
> to. This can be seen on the new Branches page [4].
>
> 4: https://prototype-guix-data-service.cbaines.net/branches
This is really nice.
This information could also be gathered directly from the repo though,
right?
I would expect only patch submission info, and possibly commit
notifications, to be grabbed from email, while the rest would be
extracted from the repo, thereby hopefully limiting the risk of
misinterpreting email. WDYT?
> There's now a basic search function on the packages page [5], and the
> location, and the licenses for packages is now being stored (which can
> be seen on the page for a package, for example [6]).
>
> 5: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
> 6: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha
Nice!
One thing that be great is a page similar to
<https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/shepherd/0.6.1>,
but keyed by package, where you get a list of the recent package
versions (and/or derivations) and map them to specific commits.
> The URL is a bit long, but I think that is now close to being possible
> with the Guix data service. I haven't got something working yet to
> easily access data for the latest revision, but for a particular
> revision, you can request a JSON file containing all the information I
> think Repology currently gets about all packages. For example:
>
> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
Awesome. (I advise passing “limit_results=900” though, because the URL
above gives a pretty big result. ;-))
> This is just the software side of the problem though. If this was to be
> used by Repology, it would have to be a more permanent thing, similar to
> the Cuirass and Mumi services that are currently setup around Guix. Does
> anyone have any thoughts on this?
I’d suggest having a Guix service for the whole thing, and making a
branch in guix-maintenance.git such that bayfront (say) can run the
service.
Then we’ll have to reach consensus on guix-sysadmin as to which machine
to use depending on the resources it needs, but if you have the config,
I’d argue that we can happily run it on bayfront or perhaps berlin. And
we can give you access to the machine so you can reconfigure once in a
while.
WDYT?
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: More progress with the Guix Data Service
2019-05-17 18:26 ` Christopher Baines
@ 2019-05-20 15:23 ` Ludovic Courtès
0 siblings, 0 replies; 7+ messages in thread
From: Ludovic Courtès @ 2019-05-20 15:23 UTC (permalink / raw)
To: Christopher Baines; +Cc: guix-devel, Guix-devel
Christopher Baines <mail@cbaines.net> skribis:
>>>
>>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
>>
>> FWIW it is very slow.
This particular request is asking for a *lot* of data, and I guess
IceCat takes a while to simply parse and pretty-print that data.
> To be honest, I'm actually pretty happy with the speed! I was worried
> that NGinx was going to time out and that it wasn't even going to work.
I found everything to be snappy, kudos!
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: More progress with the Guix Data Service
2019-05-20 15:20 ` Ludovic Courtès
@ 2019-05-20 20:14 ` Christopher Baines
2019-05-21 8:49 ` Ludovic Courtès
0 siblings, 1 reply; 7+ messages in thread
From: Christopher Baines @ 2019-05-20 20:14 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 5916 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
>> As well as listening to the Guix Commits mailing list for emails about
>> new revisions, more of the information in these emails is now stored, in
>> particular, the time they were sent, and the branch the email applies
>> to. This can be seen on the new Branches page [4].
>>
>> 4: https://prototype-guix-data-service.cbaines.net/branches
>
> This is really nice.
>
> This information could also be gathered directly from the repo though,
> right?
>
> I would expect only patch submission info, and possibly commit
> notifications, to be grabbed from email, while the rest would be
> extracted from the repo, thereby hopefully limiting the risk of
> misinterpreting email. WDYT?
So, currently the branch name, commit hash and date are taken from the
email. As far as I know, git branches are just pointers to commits, and
don't have any date/time associated with them. The commit date, or
author date in the commits could be stored and used, but I think these
are less interesting, and often misleading. The author date is often
quite different from the time a commit is pushed, and the commit date is
often different by some amount as well.
Currently, if you actually want to know what was the state of a
particular branch in the Guix git repository on Savannah was, at a
particular time, I think the most reliable way of checking would
probably be to check the guix-commits mailing list.
As the branch name, and commit hash both relate to the date, I don't see
that much problem with storing them.
One thing I've also been thinking about is loading in the guix-commits
mailing list archives. That would backfill the branch information, which
might be useful/interesting...
I did consider trying to access the clone of the Git repository that's
managed by the (guix inferiors) module, but I couldn't see an easy way
to do it, and as above, I'm not sure the date/time information is as
useful as what you can get from the mailing list.
>> There's now a basic search function on the packages page [5], and the
>> location, and the licenses for packages is now being stored (which can
>> be seen on the page for a package, for example [6]).
>>
>> 5: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
>> 6: https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha
>
> Nice!
>
> One thing that be great is a page similar to
> <https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/shepherd/0.6.1>,
> but keyed by package, where you get a list of the recent package
> versions (and/or derivations) and map them to specific commits.
Interesting, yeah, were you thinking of filtering that data for a
specific branch (like master or staging), or showing data for all
branches?
>> The URL is a bit long, but I think that is now close to being possible
>> with the Guix data service. I haven't got something working yet to
>> easily access data for the latest revision, but for a particular
>> revision, you can request a JSON file containing all the information I
>> think Repology currently gets about all packages. For example:
>>
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
>
> Awesome. (I advise passing “limit_results=900” though, because the URL
> above gives a pretty big result. ;-))
Well, not that big? Icecat tells me it's 12MB. Also, I've recently added
a "All results" checkbox/query parameter, so you no longer have to make
up a large number. I wanted to make it possible to get all the data as a
single file, as that could simplify processing it, but there's also some
support for pagination.
https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&all_results=on
The all results option is especially important as I've now done some
work on caching. That page should be served with a max-age of a day, it
could probably be even longer as well, as the only thing that will
change the contents is software changes. NGinx is also now caching
responses, and you can see what it's doing by looking at the
X-Cache-Status header in the response.
>> This is just the software side of the problem though. If this was to be
>> used by Repology, it would have to be a more permanent thing, similar to
>> the Cuirass and Mumi services that are currently setup around Guix. Does
>> anyone have any thoughts on this?
>
> I’d suggest having a Guix service for the whole thing, and making a
> branch in guix-maintenance.git such that bayfront (say) can run the
> service.
>
> Then we’ll have to reach consensus on guix-sysadmin as to which machine
> to use depending on the resources it needs, but if you have the config,
> I’d argue that we can happily run it on bayfront or perhaps berlin. And
> we can give you access to the machine so you can reconfigure once in a
> while.
>
> WDYT?
That all sounds really good :D
A package and service has been on my list of things to do, and I'll
hopefully sort that out in the next few weeks.
Currently I'm running it with Guix + support for isolated inferiors [1],
but I think that's something that can be made optional in the Guix Data
Service code, as initially I'd just be thinking about processing
revisions in the Guix git repository.
1: https://issues.guix.info/issue/34638
Thanks,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: More progress with the Guix Data Service
2019-05-20 20:14 ` Christopher Baines
@ 2019-05-21 8:49 ` Ludovic Courtès
0 siblings, 0 replies; 7+ messages in thread
From: Ludovic Courtès @ 2019-05-21 8:49 UTC (permalink / raw)
To: Christopher Baines; +Cc: guix-devel
Hello!
Christopher Baines <mail@cbaines.net> skribis:
> Ludovic Courtès <ludo@gnu.org> writes:
[...]
>> I would expect only patch submission info, and possibly commit
>> notifications, to be grabbed from email, while the rest would be
>> extracted from the repo, thereby hopefully limiting the risk of
>> misinterpreting email. WDYT?
>
> So, currently the branch name, commit hash and date are taken from the
> email. As far as I know, git branches are just pointers to commits, and
> don't have any date/time associated with them. The commit date, or
> author date in the commits could be stored and used, but I think these
> are less interesting, and often misleading. The author date is often
> quite different from the time a commit is pushed, and the commit date is
> often different by some amount as well.
>
> Currently, if you actually want to know what was the state of a
> particular branch in the Guix git repository on Savannah was, at a
> particular time, I think the most reliable way of checking would
> probably be to check the guix-commits mailing list.
Good point.
Alternately, we could get that information first-hand if we were running
the Git server ourselves (which I think we’ll have to do at some point
to improve our workflow, for things like commit signature checks.)
> As the branch name, and commit hash both relate to the date, I don't see
> that much problem with storing them.
>
> One thing I've also been thinking about is loading in the guix-commits
> mailing list archives. That would backfill the branch information, which
> might be useful/interesting...
>
> I did consider trying to access the clone of the Git repository that's
> managed by the (guix inferiors) module, but I couldn't see an easy way
> to do it, and as above, I'm not sure the date/time information is as
> useful as what you can get from the mailing list.
OK, makes sense.
>> One thing that be great is a page similar to
>> <https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/shepherd/0.6.1>,
>> but keyed by package, where you get a list of the recent package
>> versions (and/or derivations) and map them to specific commits.
>
> Interesting, yeah, were you thinking of filtering that data for a
> specific branch (like master or staging), or showing data for all
> branches?
As a user, I’d like to answer questions like “when was package X
upgraded in Guix?”, “which commit should I pull to obtain that old
version of the package?”, things like that.
>>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
>>
>> Awesome. (I advise passing “limit_results=900” though, because the URL
>> above gives a pretty big result. ;-))
>
> Well, not that big? Icecat tells me it's 12MB. Also, I've recently added
> a "All results" checkbox/query parameter, so you no longer have to make
> up a large number. I wanted to make it possible to get all the data as a
> single file, as that could simplify processing it, but there's also some
> support for pagination.
>
> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&all_results=on
>
> The all results option is especially important as I've now done some
> work on caching. That page should be served with a max-age of a day, it
> could probably be even longer as well, as the only thing that will
> change the contents is software changes. NGinx is also now caching
> responses, and you can see what it's doing by looking at the
> X-Cache-Status header in the response.
Sounds good!
>> I’d suggest having a Guix service for the whole thing, and making a
>> branch in guix-maintenance.git such that bayfront (say) can run the
>> service.
>>
>> Then we’ll have to reach consensus on guix-sysadmin as to which machine
>> to use depending on the resources it needs, but if you have the config,
>> I’d argue that we can happily run it on bayfront or perhaps berlin. And
>> we can give you access to the machine so you can reconfigure once in a
>> while.
>>
>> WDYT?
>
> That all sounds really good :D
>
> A package and service has been on my list of things to do, and I'll
> hopefully sort that out in the next few weeks.
Awesome.
> Currently I'm running it with Guix + support for isolated inferiors [1],
> but I think that's something that can be made optional in the Guix Data
> Service code, as initially I'd just be thinking about processing
> revisions in the Guix git repository.
OK. I need to take another look at the isolated inferior patches.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-05-21 8:49 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-05-17 7:56 More progress with the Guix Data Service Christopher Baines
2019-05-17 10:07 ` Amirouche
2019-05-17 18:26 ` Christopher Baines
2019-05-20 15:23 ` Ludovic Courtès
2019-05-20 15:20 ` Ludovic Courtès
2019-05-20 20:14 ` Christopher Baines
2019-05-21 8:49 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).