bug#70663: nss@3.99 is really hard to build

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> Before closing this bug, it would be good to understand more about how
>> this happened and from that try to think if anything can be done to
>> prevent similar issues in the future?
>>
>> At least from what I can see on the issues, the problem was introduced
>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>> [4]. Given the changes in 70662 were sent to guix-patches and then
>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>> for data.qa.guix.gnu.org to fail attempting to build nss.
>
> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.

Ah, yep.

> Since this was security sensitive, I built it on x86_64, tested it there
> to ensure that IceCat worked as expected, had others confirmed it worked
> for them on #guix then pushed.
>
> In the past, I've had more patience waiting for QA to build things, but
> since this is not guaranteed (it sometimes never happened), it seems
> reasonable to me to promptly push security fixes that were manually
> built & tested and adjust for any breakage later, if there is any.

I think pushing security fixes quickly is good, but this does set a
precedent on architecture support (only x86_64-linux matters).

For some packages (including nss in this instance), not looking at non
x86_64-linux support doesn't just affect users, the data service and
ci.guix.gnu.org were particularly affected, so for some packages it's
important to test across the "supported" systems just to ensure the
projects own tooling doesn't break.

>> 3: https://issues.guix.gnu.org/70662
>> 4: https://issues.guix.gnu.org/70618
>>
>> Had the changes waited for longer, then these failures should have been
>> spotted by QA, I would guess that the revision might have failed to be
>> processed, and if it was processed successfully, the nss failures should
>> have shown up, so maybe we should start requiring [5] that not only are
>> changes sent to guix-patches@gnu.org, but that QA processes them (to
>> some extent) before merging?
>
> I have some apprehensions about that; given the QA build farm is
> somewhat under-resourced for builds, I fear security changes could be
> gated for longer periods of time than is reasonable to wait.  If we go
> that route, I think we should dedicate more hardware first.

I think that's reasonable, I have been putting time in to the hardware,
but it's not been particularly easy going. The data service instances
are also still stuck on hardware I'm renting as well. In terms of QA
speed, there's the resources for data.qa.guix.gnu.org and there's the
hardware available for the bordeaux build farm.

There's also the potential to try and add more prioritisation. Currently
the data service prioritises branch revisions over patches, and newer
revisions more generally, but I guess it could prioritise security
related things. I'm not sure how to make that happen yet though, QA can
probably come up with the priorities, but I don't know how to convey
that to the data service.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]