bug#70663: nss@3.99 is really hard to build

bug#70663: nss@3.99 is really hard to build

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1617 bytes --]

nss@3.99 is really hard to build, it's so hard and so important that
data.guix.gnu.org is still after two days trying to process [1]. I say
so important because you have to build nss@3.99 to compute the channel
instance derivations for Guix.

1: https://data.guix.gnu.org/revision/72308f262c910977e40c2c9f350dc563c0a8437a

Looking at the next revision which has been processed [2], it's been
built on riscv64-linux as the testsuite is disabled, and it has also
built on aarch64-linux, but there's no successful build for any other
architecture.

2: https://data.guix.gnu.org/revision/9f183c3627a006e8fd3bb9708448bc05a6204e6d/package/nss/3.99.0?locale=en_US.UTF-8

I think there's two issues here, was this spotted before merging, and
what if anything can be done about this now. Where there's not a
substitute available for nss@3.99, this will affect guix pull/guix
time-machine, e.g.

  → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
  Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
  substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
  substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
  substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
   nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
  building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70663: nss@3.99 is really hard to build

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2217 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> nss@3.99 is really hard to build, it's so hard and so important that
> data.guix.gnu.org is still after two days trying to process [1]. I say
> so important because you have to build nss@3.99 to compute the channel
> instance derivations for Guix.
>
> 1: https://data.guix.gnu.org/revision/72308f262c910977e40c2c9f350dc563c0a8437a
>
> Looking at the next revision which has been processed [2], it's been
> built on riscv64-linux as the testsuite is disabled, and it has also
> built on aarch64-linux, but there's no successful build for any other
> architecture.
>
> 2: https://data.guix.gnu.org/revision/9f183c3627a006e8fd3bb9708448bc05a6204e6d/package/nss/3.99.0?locale=en_US.UTF-8
>
> I think there's two issues here, was this spotted before merging, and
> what if anything can be done about this now. Where there's not a
> substitute available for nss@3.99, this will affect guix pull/guix
> time-machine, e.g.
>
>   → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>    nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
>   building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...

Looking at the build failures for x86_64-linux, it seems that there's
just one test failure. There's a threshold of less than 5 seconds, and
it takes 5 to 7 seconds to complete. This probably isn't helped by using
faketime.

Here's an upstream bug [3] where they raised the threshold a bit, but
this isn't enough for our use case.

3: https://bugzilla.mozilla.org/show_bug.cgi?id=1835357

I've sent a patch here which increases the threshold by a lot:

  https://issues.guix.gnu.org/70693

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70663: nss@3.99 is really hard to build

[Maxim Cournoyer]

Hi Chris,

Christopher Baines <mail@cbaines.net> writes:

> nss@3.99 is really hard to build, it's so hard and so important that
> data.guix.gnu.org is still after two days trying to process [1]. I say
> so important because you have to build nss@3.99 to compute the channel
> instance derivations for Guix.

I agree that the nss test suite takes a ridiculous amount of time to run
(multiple hours on a fast machine IIRC).  Are we missing a '--fast' test
flag or something to make it run in a more reasonable amount of time?

-- 
Thanks,
Maxim

bug#70663: nss@3.99 is really hard to build

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Chris,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> nss@3.99 is really hard to build, it's so hard and so important that
>> data.guix.gnu.org is still after two days trying to process [1]. I say
>> so important because you have to build nss@3.99 to compute the channel
>> instance derivations for Guix.
>
> I agree that the nss test suite takes a ridiculous amount of time to run
> (multiple hours on a fast machine IIRC).  Are we missing a '--fast' test
> flag or something to make it run in a more reasonable amount of time?

I did read some of the all.sh script used for the tests and there is
some environment variables you can set here:

  https://github.com/nss-dev/nss/blob/master/tests/all.sh#L70-L82

It seems like there are 4 "cycles", maybe we could just run the standard
cycle or at least check how long they each take.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70663: nss@3.99 is really hard to build

[Christina O'Donnell]

Hi,

On 01/05/2024 18:14, Christopher Baines wrote:
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Hi Chris,
>>
>> Christopher Baines <mail@cbaines.net> writes:
>>
>>> nss@3.99 is really hard to build, it's so hard and so important that
>>> data.guix.gnu.org is still after two days trying to process [1]. I say
>>> so important because you have to build nss@3.99 to compute the channel
>>> instance derivations for Guix.
>> I agree that the nss test suite takes a ridiculous amount of time to run
>> (multiple hours on a fast machine IIRC).  Are we missing a '--fast' test
>> flag or something to make it run in a more reasonable amount of time?
> I did read some of the all.sh script used for the tests and there is
> some environment variables you can set here:
>
>    https://github.com/nss-dev/nss/blob/master/tests/all.sh#L70-L82
>
> It seems like there are 4 "cycles", maybe we could just run the standard
> cycle or at least check how long they each take.

On my machine building natively on x86_64 I was getting approximately 63 
mins for a full test and 20 mins for just the 'standard' 'cycle'. My 
vote would be to just run 'standard' since that runs all of the tests once.

I can profile individual tests if needed to see if there's any that are 
particularly worth culling, but just the above change is an easy win 
without sacrificing too much test coverage.

Kind regards,

Christina

bug#70663: nss@3.99 is really hard to build

[Joshua Branson via Bug reports for GNU Guix]

Perhaps we could disable the test suite for power9 ?  At the moment guix
pull fails on power9...I believe due to this bug.

Just a thought.

Joshua

bug#70663: nss@3.99 is really hard to build

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3164 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> nss@3.99 is really hard to build, it's so hard and so important that
> data.guix.gnu.org is still after two days trying to process [1]. I say
> so important because you have to build nss@3.99 to compute the channel
> instance derivations for Guix.
>
> 1: https://data.guix.gnu.org/revision/72308f262c910977e40c2c9f350dc563c0a8437a
>
> Looking at the next revision which has been processed [2], it's been
> built on riscv64-linux as the testsuite is disabled, and it has also
> built on aarch64-linux, but there's no successful build for any other
> architecture.
>
> 2: https://data.guix.gnu.org/revision/9f183c3627a006e8fd3bb9708448bc05a6204e6d/package/nss/3.99.0?locale=en_US.UTF-8
>
> I think there's two issues here, was this spotted before merging, and
> what if anything can be done about this now. Where there's not a
> substitute available for nss@3.99, this will affect guix pull/guix
> time-machine, e.g.
>
>   → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>    nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
>   building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...

So with the changes in #70693 merged, this issue should be fixed going
forward, but the revisions with the broken nss are going to be affected
forever and thus the impact is going to drag on for a while. For
example, data.guix.gnu.org is going to be struggling to process the
revisions with the broken nss for a long while to come.

Before closing this bug, it would be good to understand more about how
this happened and from that try to think if anything can be done to
prevent similar issues in the future?

At least from what I can see on the issues, the problem was introduced
with the update to 3.98.0 [3] and then continued with the update to 3.99
[4]. Given the changes in 70662 were sent to guix-patches and then
merged less than 24 hours later, I'd imagine that wasn't sufficient time
for data.qa.guix.gnu.org to fail attempting to build nss.

3: https://issues.guix.gnu.org/70662
4: https://issues.guix.gnu.org/70618

Had the changes waited for longer, then these failures should have been
spotted by QA, I would guess that the revision might have failed to be
processed, and if it was processed successfully, the nss failures should
have shown up, so maybe we should start requiring [5] that not only are
changes sent to guix-patches@gnu.org, but that QA processes them (to
some extent) before merging?

5: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html#

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70663: nss@3.99 is really hard to build

[pelzflorian (Florian Pelz)]

Hello Christopher.

Christopher Baines <mail@cbaines.net> writes:
> Had the changes waited for longer, then these failures should have been
> spotted by QA, I would guess that the revision might have failed to be
> processed, and if it was processed successfully, the nss failures should
> have shown up, so maybe we should start requiring [5] that not only are
> changes sent to guix-patches@gnu.org, but that QA processes them (to
> some extent) before merging?
>
> 5: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html#

Yes, though note that the nss change did provide security fixes:

commit e584ff08b162c46ef587daca438e97d56bc20b32
Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Date:   Wed Apr 24 11:22:30 2024 -0400

    gnu: nss: Graft with version 3.98 [security fixes].
    
    This fixes CVE-2023-5388, CVE-2023-6135 and CVE-2024-0743.
    
    * gnu/packages/nss.scm (nss) [replacement]: New field.
    (nss-3.98): Rename variable to...
    (nss/fixed): ... this.  Make it a hidden package.
    * gnu/packages/librewolf.scm (librewolf) [inputs]: Replace nss-3.98 with
    nss/fixed.
    
    Change-Id: I8cc667c53a270dfe00738bf731923f1342036624

I suppose the requirement to wait for QA should apply to security fixes
as well?

Thank you for all your work.

Regards,
Florian

bug#70663: nss@3.99 is really hard to build

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1729 bytes --]

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> writes:

> Hello Christopher.
>
> Christopher Baines <mail@cbaines.net> writes:
>> Had the changes waited for longer, then these failures should have been
>> spotted by QA, I would guess that the revision might have failed to be
>> processed, and if it was processed successfully, the nss failures should
>> have shown up, so maybe we should start requiring [5] that not only are
>> changes sent to guix-patches@gnu.org, but that QA processes them (to
>> some extent) before merging?
>>
>> 5: https://guix.gnu.org/manual/devel/en/html_node/Managing-Patches-and-Branches.html#
>
> Yes, though note that the nss change did provide security fixes:
>
> commit e584ff08b162c46ef587daca438e97d56bc20b32
> Author: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date:   Wed Apr 24 11:22:30 2024 -0400
>
>     gnu: nss: Graft with version 3.98 [security fixes].
>
>     This fixes CVE-2023-5388, CVE-2023-6135 and CVE-2024-0743.
>
>     * gnu/packages/nss.scm (nss) [replacement]: New field.
>     (nss-3.98): Rename variable to...
>     (nss/fixed): ... this.  Make it a hidden package.
>     * gnu/packages/librewolf.scm (librewolf) [inputs]: Replace nss-3.98 with
>     nss/fixed.
>
>     Change-Id: I8cc667c53a270dfe00738bf731923f1342036624
>
> I suppose the requirement to wait for QA should apply to security fixes
> as well?

Well, there's a risk in not testing things across multiple
machines/architectures at least. The value of getting a security fix
merged quickly is reduced if users on some architectures/systems can't
use it.

There's always going to be trade offs, and that's fine, but the question
is more what can be done to try and improve things for the future.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70663: nss@3.99 is really hard to build

[Maxim Cournoyer]

Hi,

Christopher Baines <mail@cbaines.net> writes:

[...]

>> I think there's two issues here, was this spotted before merging, and
>> what if anything can be done about this now. Where there's not a
>> substitute available for nss@3.99, this will affect guix pull/guix
>> time-machine, e.g.
>>
>>   → guix time-machine --commit=72308f262c910977e40c2c9f350dc563c0a8437a -- describe
>>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>>   substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
>>    nss-3.99.tar.xz  55.2MiB                                                                                             13.7MiB/s 00:04 ▕██████████████████▏ 100.0%
>>   building /gnu/store/8379qa0y6s7ssjr8gplm5fyw9r5pnxhn-nss-3.99.0.drv...
>
> So with the changes in #70693 merged, this issue should be fixed going
> forward, but the revisions with the broken nss are going to be affected
> forever and thus the impact is going to drag on for a while. For
> example, data.guix.gnu.org is going to be struggling to process the
> revisions with the broken nss for a long while to come.
>
> Before closing this bug, it would be good to understand more about how
> this happened and from that try to think if anything can be done to
> prevent similar issues in the future?
>
> At least from what I can see on the issues, the problem was introduced
> with the update to 3.98.0 [3] and then continued with the update to 3.99
> [4]. Given the changes in 70662 were sent to guix-patches and then
> merged less than 24 hours later, I'd imagine that wasn't sufficient time
> for data.qa.guix.gnu.org to fail attempting to build nss.

I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.

Since this was security sensitive, I built it on x86_64, tested it there
to ensure that IceCat worked as expected, had others confirmed it worked
for them on #guix then pushed.

In the past, I've had more patience waiting for QA to build things, but
since this is not guaranteed (it sometimes never happened), it seems
reasonable to me to promptly push security fixes that were manually
built & tested and adjust for any breakage later, if there is any.

> 3: https://issues.guix.gnu.org/70662
> 4: https://issues.guix.gnu.org/70618
>
> Had the changes waited for longer, then these failures should have been
> spotted by QA, I would guess that the revision might have failed to be
> processed, and if it was processed successfully, the nss failures should
> have shown up, so maybe we should start requiring [5] that not only are
> changes sent to guix-patches@gnu.org, but that QA processes them (to
> some extent) before merging?

I have some apprehensions about that; given the QA build farm is
somewhat under-resourced for builds, I fear security changes could be
gated for longer periods of time than is reasonable to wait.  If we go
that route, I think we should dedicate more hardware first.

-- 
Thanks,
Maxim

bug#70663: nss@3.99 is really hard to build

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> Before closing this bug, it would be good to understand more about how
>> this happened and from that try to think if anything can be done to
>> prevent similar issues in the future?
>>
>> At least from what I can see on the issues, the problem was introduced
>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>> [4]. Given the changes in 70662 were sent to guix-patches and then
>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>> for data.qa.guix.gnu.org to fail attempting to build nss.
>
> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.

Ah, yep.

> Since this was security sensitive, I built it on x86_64, tested it there
> to ensure that IceCat worked as expected, had others confirmed it worked
> for them on #guix then pushed.
>
> In the past, I've had more patience waiting for QA to build things, but
> since this is not guaranteed (it sometimes never happened), it seems
> reasonable to me to promptly push security fixes that were manually
> built & tested and adjust for any breakage later, if there is any.

I think pushing security fixes quickly is good, but this does set a
precedent on architecture support (only x86_64-linux matters).

For some packages (including nss in this instance), not looking at non
x86_64-linux support doesn't just affect users, the data service and
ci.guix.gnu.org were particularly affected, so for some packages it's
important to test across the "supported" systems just to ensure the
projects own tooling doesn't break.

>> 3: https://issues.guix.gnu.org/70662
>> 4: https://issues.guix.gnu.org/70618
>>
>> Had the changes waited for longer, then these failures should have been
>> spotted by QA, I would guess that the revision might have failed to be
>> processed, and if it was processed successfully, the nss failures should
>> have shown up, so maybe we should start requiring [5] that not only are
>> changes sent to guix-patches@gnu.org, but that QA processes them (to
>> some extent) before merging?
>
> I have some apprehensions about that; given the QA build farm is
> somewhat under-resourced for builds, I fear security changes could be
> gated for longer periods of time than is reasonable to wait.  If we go
> that route, I think we should dedicate more hardware first.

I think that's reasonable, I have been putting time in to the hardware,
but it's not been particularly easy going. The data service instances
are also still stuck on hardware I'm renting as well. In terms of QA
speed, there's the resources for data.qa.guix.gnu.org and there's the
hardware available for the bordeaux build farm.

There's also the potential to try and add more prioritisation. Currently
the data service prioritises branch revisions over patches, and newer
revisions more generally, but I guess it could prioritise security
related things. I'm not sure how to make that happen yet though, QA can
probably come up with the priorities, but I don't know how to convey
that to the data service.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70663: nss@3.99 is really hard to build

[Mark H Weaver]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Christopher Baines <mail@cbaines.net> writes:
[...]
>> At least from what I can see on the issues, the problem was introduced
>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>> [4]. Given the changes in 70662 were sent to guix-patches and then
>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>> for data.qa.guix.gnu.org to fail attempting to build nss.
>
> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.
>
> Since this was security sensitive, I built it on x86_64, tested it there
> to ensure that IceCat worked as expected, had others confirmed it worked
> for them on #guix then pushed.
[...]
>> 3: https://issues.guix.gnu.org/70662
>> 4: https://issues.guix.gnu.org/70618

Note that the IceCat package in Guix currently uses the copy of NSS that
comes bundled with the IceCat source code, so testing IceCat probably
won't tell you much about whether the standalone NSS package in Guix
works properly.

     Regards,
       Mark

bug#70663: nss@3.99 is really hard to build

[Maxim Cournoyer]

Hi Mark,

Mark H Weaver <mhw@netris.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Christopher Baines <mail@cbaines.net> writes:
> [...]
>>> At least from what I can see on the issues, the problem was introduced
>>> with the update to 3.98.0 [3] and then continued with the update to 3.99
>>> [4]. Given the changes in 70662 were sent to guix-patches and then
>>> merged less than 24 hours later, I'd imagine that wasn't sufficient time
>>> for data.qa.guix.gnu.org to fail attempting to build nss.
>>
>> I think in [3] you meant https://issues.guix.gnu.org/70569, not #70662.
>>
>> Since this was security sensitive, I built it on x86_64, tested it there
>> to ensure that IceCat worked as expected, had others confirmed it worked
>> for them on #guix then pushed.
> [...]
>>> 3: https://issues.guix.gnu.org/70662
>>> 4: https://issues.guix.gnu.org/70618
>
> Note that the IceCat package in Guix currently uses the copy of NSS that
> comes bundled with the IceCat source code, so testing IceCat probably
> won't tell you much about whether the standalone NSS package in Guix
> works properly.

Thanks for the heads-up.  It looks like there are now some low hanging
fruits in terms of unbundling opportunities for icecat/Icedove!

-- 
Thanks,
Maxim

bug#70663: nss@3.99 is really hard to build

[Ian Eure]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Mark,
>
> Mark H Weaver <mhw@netris.org> writes:
>
>> Hi Maxim,
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>
>>> Christopher Baines <mail@cbaines.net> writes:
>> [...]
>>>> At least from what I can see on the issues, the problem was 
>>>> introduced
>>>> with the update to 3.98.0 [3] and then continued with the 
>>>> update to 3.99
>>>> [4]. Given the changes in 70662 were sent to guix-patches and 
>>>> then
>>>> merged less than 24 hours later, I'd imagine that wasn't 
>>>> sufficient time
>>>> for data.qa.guix.gnu.org to fail attempting to build nss.
>>>
>>> I think in [3] you meant https://issues.guix.gnu.org/70569, 
>>> not #70662.
>>>
>>> Since this was security sensitive, I built it on x86_64, 
>>> tested it there
>>> to ensure that IceCat worked as expected, had others confirmed 
>>> it worked
>>> for them on #guix then pushed.
>> [...]
>>>> 3: https://issues.guix.gnu.org/70662
>>>> 4: https://issues.guix.gnu.org/70618
>>
>> Note that the IceCat package in Guix currently uses the copy of 
>> NSS that
>> comes bundled with the IceCat source code, so testing IceCat 
>> probably
>> won't tell you much about whether the standalone NSS package in 
>> Guix
>> works properly.
>
> Thanks for the heads-up.  It looks like there are now some low 
> hanging
> fruits in terms of unbundling opportunities for icecat/Icedove!
>

Definitely.  The LibreWolf package is probably a good reference, 
as I was able to unbundle all its library dependencies and use the 
Guix-packaged versions instead.

  — Ian