Re: More progress with the Guix Data Service

["Ludovic Courtès" <ludo@gnu.org>]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> I would expect only patch submission info, and possibly commit
>> notifications, to be grabbed from email, while the rest would be
>> extracted from the repo, thereby hopefully limiting the risk of
>> misinterpreting email.  WDYT?
>
> So, currently the branch name, commit hash and date are taken from the
> email. As far as I know, git branches are just pointers to commits, and
> don't have any date/time associated with them. The commit date, or
> author date in the commits could be stored and used, but I think these
> are less interesting, and often misleading. The author date is often
> quite different from the time a commit is pushed, and the commit date is
> often different by some amount as well.
>
> Currently, if you actually want to know what was the state of a
> particular branch in the Guix git repository on Savannah was, at a
> particular time, I think the most reliable way of checking would
> probably be to check the guix-commits mailing list.

Good point.

Alternately, we could get that information first-hand if we were running
the Git server ourselves (which I think we’ll have to do at some point
to improve our workflow, for things like commit signature checks.)

> As the branch name, and commit hash both relate to the date, I don't see
> that much problem with storing them.
>
> One thing I've also been thinking about is loading in the guix-commits
> mailing list archives. That would backfill the branch information, which
> might be useful/interesting...
>
> I did consider trying to access the clone of the Git repository that's
> managed by the (guix inferiors) module, but I couldn't see an easy way
> to do it, and as above, I'm not sure the date/time information is as
> useful as what you can get from the mailing list.

OK, makes sense.

>> One thing that be great is a page similar to
>> <https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/shepherd/0.6.1>,
>> but keyed by package, where you get a list of the recent package
>> versions (and/or derivations) and map them to specific commits.
>
> Interesting, yeah, were you thinking of filtering that data for a
> specific branch (like master or staging), or showing data for all
> branches?

As a user, I’d like to answer questions like “when was package X
upgraded in Guix?”, “which commit should I pull to obtain that old
version of the package?”, things like that.

>>>   https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
>>
>> Awesome.  (I advise passing “limit_results=900” though, because the URL
>> above gives a pretty big result.  ;-))
>
> Well, not that big? Icecat tells me it's 12MB. Also, I've recently added
> a "All results" checkbox/query parameter, so you no longer have to make
> up a large number. I wanted to make it possible to get all the data as a
> single file, as that could simplify processing it, but there's also some
> support for pagination.
>
>   https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&all_results=on
>
> The all results option is especially important as I've now done some
> work on caching. That page should be served with a max-age of a day, it
> could probably be even longer as well, as the only thing that will
> change the contents is software changes. NGinx is also now caching
> responses, and you can see what it's doing by looking at the
> X-Cache-Status header in the response.

Sounds good!

>> I’d suggest having a Guix service for the whole thing, and making a
>> branch in guix-maintenance.git such that bayfront (say) can run the
>> service.
>>
>> Then we’ll have to reach consensus on guix-sysadmin as to which machine
>> to use depending on the resources it needs, but if you have the config,
>> I’d argue that we can happily run it on bayfront or perhaps berlin.  And
>> we can give you access to the machine so you can reconfigure once in a
>> while.
>>
>> WDYT?
>
> That all sounds really good :D
>
> A package and service has been on my list of things to do, and I'll
> hopefully sort that out in the next few weeks.

Awesome.

> Currently I'm running it with Guix + support for isolated inferiors [1],
> but I think that's something that can be made optional in the Guix Data
> Service code, as initially I'd just be thinking about processing
> revisions in the Guix git repository.

OK.  I need to take another look at the isolated inferior patches.

Thanks,
Ludo’.