core-updates-frozen: Planning for the last world rebuild

core-updates-frozen: Planning for the last world rebuild

[Ludovic Courtès]

Hello Guix!

Wouldn’t it be nice to get ‘core-updates-frozen’ merged?  :-)

What about merging the patches blocked by
<https://issues.guix.gnu.org/50358>?  We could even have them on a
separate branch, have Cuirass build it, and merge it 24–36h later, at
which point substitute coverage should be reasonably good.

I haven’t followed the new ports, POWER9 and RISC-V.  If there’s
anything needed in this area, like localized toolchain changes, please
send a heads-up!

As I say every time someone proposes a new port, the main issue is not
the port itself but maintaining it longer-term.  And here it’s not even
clear to me what the status of POWER9 is on ‘core-updates-frozen’, so
it’d be great to get feedback on this, ensure Cuirass builds it, and so
on.

Thanks,
Ludo’.

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey!

> Wouldn’t it be nice to get ‘core-updates-frozen’ merged?  :-)

That would be great :) Before merging to master, it would be nice to
have more system tests passing. Currently 16 out of 87 are
broken. Except for the nfs-root-fs that has always been broken, the
other tests probably cover user configurations out there.

I'm currently working on the installation tests.

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[John Kehayias]

Hi Ludo’ and Guixers,

Thanks for the good progress on core-updates-frozen (where I'm currently writing from and have nearly everything working as before). Seems like we have a few different bug numbers and threads for the world rebuild. Sorry for repeating from #50860, which I will paste below.

In summary, I also noticed an older bug for Flatpak with p11-kit, as well as p11-kit being out of date. Locally I've tested it builds with the patch and version update and fixes the bug. The configuration change is the same as nix does for p11-kit for the same reason, for what that is worth.

Less critical is Mesa continuing to put out updates, both on 21.1 and now moving to 21.2 (considered stable I believe). I thought we might sneak that in if we're doing lots of rebuilding.

Thanks! Original message with more details below:

Is there anything else with huge rebuilds to push together? I don't want to keep finding and adding things, but two possibilities come to mind that I've just noticed:

1a. p11-kit #49957 https://issues.guix.gnu.org/49957

I've just hit this bug on core-updates-frozen as well, though was originally reported on master. As I noted there, I tried to test with just grafts but didn't fix it for me (I'm guessing grafting won't work with that configure flag change). The patch matches how nix configures p11-kit as well, due to this bug.

1b. p11-kit is also now out of date. The changelog doesn't look substantial or serious, but given the nature of p11-kit I wonder if we want to update it now too. https://github.com/p11-glue/p11-kit/releases/tag/0.24.0

2. (minor) Mesa has had a few more bugfix and major releases since my initial patch for core-updates-frozen. Now at 21.1.8 for the 21.1 branch (we have 21.1.6 currently), but 21.2 has also had stable releases, with 21.2.2. I previously built 21.2.1 and sent a patch for it, and could test 21.2.2 if we want to do that too.

I'm aware this could continue forever, and #2 is likely lower priority. #1, though, should we consider p11-kit a critical update (version, at least) since we're still fixing core-updates-frozen?

John

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey!

> have more system tests passing. Currently 16 out of 87 are
> broken. Except for the nfs-root-fs that has always been broken, the
> other tests probably cover user configurations out there.

Made some progress here but there are still a bit more work needed:

* childhurd (https://ci.guix.gnu.org/build/901219/details)
 => gnumach build appears to be broken.

* dovecot (https://ci.guix.gnu.org/build/901189/details)
 => dovecot build appears to be broken.

* patchwork (https://ci.guix.gnu.org/build/901227/details)
 => python-django-pipeline build (dependency of patchwork) appears to be
  broken.

* ganeti-kvm (https://ci.guix.gnu.org/build/901158/details)
 => ganeti build appears to be broken.

* ganeti-lxc (https://ci.guix.gnu.org/build/901157/details)
 => same reason.

* getmail (https://ci.guix.gnu.org/build/901187/details)
 => dovecot build appears to be broken.

* gui-installed-desktop-os-encrypted
  (https://ci.guix.gnu.org/build/923078/details)
 => emacs-exwm build is broken presumably because of
  https://issues.guix.gnu.org/50066.

* jami-provisioning (https://ci.guix.gnu.org/build/901214/details) and
 jami (https://ci.guix.gnu.org/build/901213/details)
 => sobjectizer build appears to be broken.

* ldap (https://ci.guix.gnu.org/build/901179/details)
 => 389-ds-base build appears to be broken.

The overall status can be seen here:
https://ci.guix.gnu.org/eval/24933/dashboard. Let's get those tests fixed
so that we can start this branch on good grounds!

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Thiago Jung Bauermann]

Hello Ludo,

Em terça-feira, 28 de setembro de 2021, às 08:57:45 -03, Ludovic Courtès 
escreveu:
> Hello Guix!
> 
> Wouldn’t it be nice to get ‘core-updates-frozen’ merged?  :-)
> 
> What about merging the patches blocked by
> <https://issues.guix.gnu.org/50358>?  We could even have them on a
> separate branch, have Cuirass build it, and merge it 24–36h later, at
> which point substitute coverage should be reasonably good.
> 
> I haven’t followed the new ports, POWER9 and RISC-V.  If there’s
> anything needed in this area, like localized toolchain changes, please
> send a heads-up!

There are two things I’m aware of for POWER9:

• diffutils test fix: https://issues.guix.gnu.org/50239
• gcc-5 build with GCC 10 fix: https://issues.guix.gnu.org/49896

In the case of diffutils, IMHO ideally we would apply the v2 patch there for 
the world rebuild, but as Ludo suggested it’s also possible to make the 
change only for powerpc64le-linux and avoid affecting other platforms.

In the case of gcc-5, it’s the same situation actually. But in addition to 
that, `guix refresh --list-dependent` says that modifying gcc-5 would only 
cause 32 dependent packages to be rebuilt so it’s not necessary to address 
it for the world rebuild.

> As I say every time someone proposes a new port, the main issue is not
> the port itself but maintaining it longer-term.  And here it’s not even
> clear to me what the status of POWER9 is on ‘core-updates-frozen’, so
> it’d be great to get feedback on this, ensure Cuirass builds it, and so
> on.

Yeah, the status of POWER9 on ‘core-updates-frozen’ is obfuscated by some 
build failures early in the dependency chain that were caused by QEMU bugs.
My impression is that the same is also true for aarch64-linux. For 
instance, QEMU can’t run libsigsegv’s testsuite (among other things), and 
that blocks a ton of stuff. Ditto for util-linux’s testsuite.

I was waiting for this world rebuild to happen to see the results in 
Cuirass now that powerpc64le-linux builds are only done on guixp9, but in 
hindsight it would have been better if I requested that those builds were 
restarted instead. Restarting this build should remove the main roadblock 
though:

https://ci.guix.gnu.org/build/703194/details

When that build is restarted and succeeds, does Cuirass also restart the 
builds that failed because they depended on it, or do we need to explicitly 
restart those as well?

What I can say is that IMHO POWER9 support isn’t too bad at least for the 
base packages given that I can build e.g., emacs (assuming that the patch 
for issue 50521 fixing a problem in gtk+-2 is applied), which depends on 
many base packages including X11 and GTK+.

-- 
Thanks,
Thiago

Re: core-updates-frozen: Planning for the last world rebuild

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

On Tue, Sep 28, 2021 at 01:57:45PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> Wouldn’t it be nice to get ‘core-updates-frozen’ merged?  :-)
> 
> What about merging the patches blocked by
> <https://issues.guix.gnu.org/50358>?  We could even have them on a
> separate branch, have Cuirass build it, and merge it 24–36h later, at
> which point substitute coverage should be reasonably good.
> 
> I haven’t followed the new ports, POWER9 and RISC-V.  If there’s
> anything needed in this area, like localized toolchain changes, please
> send a heads-up!
> 
> As I say every time someone proposes a new port, the main issue is not
> the port itself but maintaining it longer-term.  And here it’s not even
> clear to me what the status of POWER9 is on ‘core-updates-frozen’, so
> it’d be great to get feedback on this, ensure Cuirass builds it, and so
> on.

I'm still building away on riscv64-linux. I have some minor changes,
namely renaming target-riscv? to target-riscv64? As far as cuirass
support I have it officially unsupported in %hydra-supported-systems (or
whatever that variable is called). I've tried to keep the patches so
that they can be applied without interfering with the other
architectures. At this point as far as I know it's waiting for another
pair of eyes or so to say it looks OK to merge.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey Thiago,

> I was waiting for this world rebuild to happen to see the results in 
> Cuirass now that powerpc64le-linux builds are only done on guixp9, but in 
> hindsight it would have been better if I requested that those builds were 
> restarted instead. Restarting this build should remove the main roadblock 
> though:
>
> https://ci.guix.gnu.org/build/703194/details

Now restarted!

>
> When that build is restarted and succeeds, does Cuirass also restart the 
> builds that failed because they depended on it, or do we need to explicitly 
> restart those as well?

Yes, on successful build completion Cuirass looks if some builds that
were marked as "failed-dependency" (failing because some of the
dependencies are also failing) can be resumed.

That being said, Cuirass is not performing nicely these days because of:
this issue https://issues.guix.gnu.org/48468.

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Thiago Jung Bauermann]

Hello Mathieu,

Em quinta-feira, 30 de setembro de 2021, às 06:03:52 -03, Mathieu Othacehe 
escreveu:
> Hey Thiago,
> 
> > I was waiting for this world rebuild to happen to see the results in
> > Cuirass now that powerpc64le-linux builds are only done on guixp9, but
> > in hindsight it would have been better if I requested that those
> > builds were restarted instead. Restarting this build should remove the
> > main roadblock though:
> > 
> > https://ci.guix.gnu.org/build/703194/details
> 
> Now restarted!

Thank you! Unfortunately that build is now affected by an issue that I only 
noticed yesterday: powerpc64le-linux builds for core-updates-frozen get 
permanently stuck in the ‘Scheduled’ state, even though guixp9 is 
completely idle (as it is right now).

For example, this evaluation from Sep 17 has thousands of scheduled 
evaluations:

https://ci.guix.gnu.org/eval/19271?status=pending

This is also happening with at least one other platform as well. This 
evaluation from Sep 25 has a lot of scheduled evaluations for i586-gnu (as 
well as powerpc64le-linux):

https://ci.guix.gnu.org/eval/23187?status=pending

> > When that build is restarted and succeeds, does Cuirass also restart
> > the
> > builds that failed because they depended on it, or do we need to
> > explicitly restart those as well?
> 
> Yes, on successful build completion Cuirass looks if some builds that
> were marked as "failed-dependency" (failing because some of the
> dependencies are also failing) can be resumed.

That is awesome!

> That being said, Cuirass is not performing nicely these days because of:
> this issue https://issues.guix.gnu.org/48468.

Ah, ok. Thank you for the heads up.

-- 
Thanks,
Thiago

Re: core-updates-frozen: Planning for the last world rebuild

[Ricardo Wurmus]

Mathieu Othacehe <othacehe@gnu.org> writes:

> * ldap (https://ci.guix.gnu.org/build/901179/details)
>  => 389-ds-base build appears to be broken.

I fixed the build.  Also had to upgrade 389-ds-base to get around 
a linker error.

-- 
Ricardo

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey,

> For example, this evaluation from Sep 17 has thousands of scheduled 
> evaluations:
>
> https://ci.guix.gnu.org/eval/19271?status=pending

Mmmh, not nice. Any chance you could share the cuirass-remote-worker
logs on the guixp9 machine?

Thanks,

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hello Ricardo,

> I fixed the build.  Also had to upgrade 389-ds-base to get around a linker
> error.

Thanks for your support here! The ldap test is now run, but sadly there
are some failures: https://ci.guix.gnu.org/build/928047/log/raw.

Mathieu

Re: core-updates-frozen: Planning for the last world rebuild

[Ricardo Wurmus]

Hi Mathieu,

>> I fixed the build.  Also had to upgrade 389-ds-base to get 
>> around a linker
>> error.
>
> Thanks for your support here! The ldap test is now run, but 
> sadly there
> are some failures: https://ci.guix.gnu.org/build/928047/log/raw.

Thanks for checking!

It should be fine now.  I didn’t run the test myself, but I ran 
most of the commands and found a few more hardcoded tool file 
names that referenced /sbin or /usr/bin. 

-- 
Ricardo

Re: core-updates-frozen: Planning for the last world rebuild

[Thiago Jung Bauermann]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

Em sexta-feira, 1 de outubro de 2021, às 03:21:16 -03, Mathieu Othacehe 
escreveu:
> Hey,
> 
> > For example, this evaluation from Sep 17 has thousands of scheduled
> > evaluations:
> > 
> > https://ci.guix.gnu.org/eval/19271?status=pending
> 
> Mmmh, not nice. Any chance you could share the cuirass-remote-worker
> logs on the guixp9 machine?

Of course. Here it is. For some reason there’s no log before Sep 14, but at 
least it covers the Cuirass evaluations I mentioned.

I restarted the Cuirass worker service a couple of times to adjust the 
number of workers. Perhaps it is related? I tried to do it at times when 
guixp9 wasn’t running CI jobs.

-- 
Thanks,
Thiago

[-- Attachment #2: cuirass.log.xz --]
[-- Type: application/x-xz, Size: 161508 bytes --]

Re: core-updates-frozen: Planning for the last world rebuild

[Mathieu Othacehe]

Hey Ricardo,

> It should be fine now.  I didn’t run the test myself, but I ran most of the
> commands and found a few more hardcoded tool file names that referenced /sbin
> or /usr/bin. 

Yes, I confirm it passes! I also fixed the childhurd test. Only 9 tests
to go :).

Thanks,

Mathieu