Need people to help with kernel updates

Need people to help with kernel updates

[Leo Famulari]

Hello,

For a few years, I've been handling updates of the linux-libre kernel by
myself. Now I want some more people to help with this.

The work itself is fairly mechanical and updates occur about once a
week. It takes about 30 minutes to prepare the patches and push them to
CI or send them to the mailing list.

When a new major kernel version is released, then we also need to make a
new kernel config file, which takes a few hours in total.

There is plenty of support for the CI and QA infrastructure to build the
kernels, so you don't need a powerful computer.

This isn't the sort of the task that needs to be performed by a single
person. The work could be spread, like most other packages in Guix.

If you want to join in, please reply!

Leo

Re: Need people to help with kernel updates

[Wilko Meyer]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> For a few years, I've been handling updates of the linux-libre kernel by
> myself.

First of all: thanks for doing this!

> The work itself is fairly mechanical and updates occur about once a
> week. It takes about 30 minutes to prepare the patches and push them to
> CI or send them to the mailing list.

I could imagine myself helping with these tasks. Practically this means,
that, whenever a new linux-libre minor update is being released, the
versions in linux-libre-* packages in gnu/packages/linux.scm have to be
bumped/a patch has to be sent?

Also: Is there anything to know/to have in mind when generating a new
kernel config for major releases?

> There is plenty of support for the CI and QA infrastructure to build the
> kernels, so you don't need a powerful computer.

How's the coverage for different ISA? Do the current CI jobs also cover
all the architectures we seem to support:

'("x86_64-linux" "i686-linux" "armhf-linux"
    "aarch64-linux" "powerpc64le-linux" "riscv64-linux")

or are there cases where it could be beneficial to build locally first
using:

guix build -s $ISA linux-libre

for certain ISAs? I could use my workstation for this, if there's a
benefit to it.

> If you want to join in, please reply!

How would the communication around this be organized? If n>=2 people are
trying to work on the same set of tasks duplication may happen.

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu

Re: Need people to help with kernel updates

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 4109 bytes --]

On Sat, Oct 07, 2023 at 08:31:58PM +0200, Wilko Meyer wrote:
> I could imagine myself helping with these tasks. Practically this means,
> that, whenever a new linux-libre minor update is being released, the
> versions in linux-libre-* packages in gnu/packages/linux.scm have to be
> bumped/a patch has to be sent?

Yes, the hashes of the kernel source code and linux-libre's "deblobbing"
scripts have to be updated. I have some scripts that fetch and calculate
the hashes (attached).

> Also: Is there anything to know/to have in mind when generating a new
> kernel config for major releases?

My workflow is to check out the upstream source code, copy the previous
version's configuration file into the source tree, create a development
environment with `guix shell -D linux-libre`, and then do `make
oldconfig`.

The idea with a distro kernel configuration is to make it as generically
useful as possible. That means enabling support for all kinds of
hardware and picking sensible defaults (which are usually the upstream
default).

I use the cateee.net website to get a little more info about things I
don't understand, and use Google and LKML to go beyond if necessary.
Cateee.net has references for every kernel configuration option. For
example:

https://cateee.net/lkddb/web-lkddb/HSA_AMD.html

If I'm really puzzled I ask on IRC or the mailing list, but this is a
"best effort" kind of task. In my several years with Guix, the kernel
config has occasionally received feedback.

I'm not a kernel developer or expert. The upstream defaults for kernel
'settings' are sensible. We usually differ from the defaults by enabling
support for lots of hardware.

> How's the coverage for different ISA? Do the current CI jobs also cover
> all the architectures we seem to support:
> 
> '("x86_64-linux" "i686-linux" "armhf-linux"
>     "aarch64-linux" "powerpc64le-linux" "riscv64-linux")

There's a 'kernel-updates' jobset for ci.guix.gnu.org that builds off
the 'kernel-updates' Git branch:

https://ci.guix.gnu.org/jobset/kernel-updates
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm?id=8fa8d6affb7702fa81c795de5625c7b7938fae85#n359

And qa.guix.gnu.org will build any patches that are submitted to the
mailing list.

To be honest, I only pay attention to x86_64-linux and i686-linux. The
build farm don't reliably build the kernel packages for other
architectures (not even the source code!) on the build farm and nobody
has stepped up to make sure the kernel packages build there.

My impression is that x86_64-linux is by far the most popular platform
for Guix users, and then aarch64-linux, and then the rest.

My understanding of kernels for aarch64-linux is that the 'generic'
kernel packages work better for users than the packages built with the
Guix kernel configs, and that's fine. The 'generic' kernel configs are
generated automatically.

Architectural support is a problem of the type "What came first: the
chicken or the egg?" So, if anyone wants to improve support for these
other architectures, you'll be making an egg from scratch, in the hope
that people will start using the kernels :)

> or are there cases where it could be beneficial to build locally first
> using:
> 
> guix build -s $ISA linux-libre
> 
> for certain ISAs? I could use my workstation for this, if there's a
> benefit to it.

I don't do this, but it wouldn't hurt! I just use the resources on CI
and QA. I do build my own x86_64-linux kernels for my own machines, as a
minor "full test", but that's not expected to join in this work.

> How would the communication around this be organized? If n>=2 people are
> trying to work on the same set of tasks duplication may happen.

I invite you to choose, email or IRC :)

To summarize, this work needs regular but brief attention. There's not
much feedback from the community, so we do our best and make sure the
basics work before pushing (reboot and connect to the internet). I'm
eager to help grow the community of people working on this, and can help
answer questions and give advice about things like the configs.

Leo

[-- Attachment #2: linux-libre packaging scripts.tar.xz --]
[-- Type: application/octet-stream, Size: 33153 bytes --]

Re: Need people to help with kernel updates

[John Kehayias]

Hi Leo,


On Sat, Oct 07, 2023 at 02:04 PM, Leo Famulari wrote:

> Hello,
>
> For a few years, I've been handling updates of the linux-libre kernel by
> myself. Now I want some more people to help with this.
>

Just wanted to say thanks for this work that goes on mostly behind the
scenes; I've never had any kernel hiccups related to Guix. Let's hope
I didn't just jinx that, but our rollbacks and generation selection at
boot actually make it a lot less painful to experiment here.

John

> The work itself is fairly mechanical and updates occur about once a
> week. It takes about 30 minutes to prepare the patches and push them to
> CI or send them to the mailing list.
>
> When a new major kernel version is released, then we also need to make a
> new kernel config file, which takes a few hours in total.
>
> There is plenty of support for the CI and QA infrastructure to build the
> kernels, so you don't need a powerful computer.
>
> This isn't the sort of the task that needs to be performed by a single
> person. The work could be spread, like most other packages in Guix.
>
> If you want to join in, please reply!
>
> Leo

Re: Need people to help with kernel updates

[Luis Felipe]

[-- Attachment #1.1.1: Type: text/plain, Size: 602 bytes --]

El 9/10/23 a las 1:22, John Kehayias escribió:
> Hi Leo,
>
>
> On Sat, Oct 07, 2023 at 02:04 PM, Leo Famulari wrote:
>
>> Hello,
>>
>> For a few years, I've been handling updates of the linux-libre kernel by
>> myself. Now I want some more people to help with this.
>>
> Just wanted to say thanks for this work that goes on mostly behind the
> scenes; I've never had any kernel hiccups related to Guix. Let's hope
> I didn't just jinx that, but our rollbacks and generation selection at
> boot actually make it a lot less painful to experiment here.
>
Same here, thanks a lot, Leo :)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 2881 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Need people to help with kernel updates

[Wilko Meyer]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> Yes, the hashes of the kernel source code and linux-libre's "deblobbing"
> scripts have to be updated. I have some scripts that fetch and calculate
> the hashes (attached).

Thanks for sharing these scripts with me, I've had a look into them to
get familiar with them & so far they seem pretty useful!

> I'm not a kernel developer or expert. The upstream defaults for kernel
> 'settings' are sensible. We usually differ from the defaults by enabling
> support for lots of hardware.

Sounds reasonable, supporting as much hardware as possible seems
feasible for a generic config for a kernel build.

> My impression is that x86_64-linux is by far the most popular platform
> for Guix users, and then aarch64-linux, and then the rest.
>
> Architectural support is a problem of the type "What came first: the
> chicken or the egg?" So, if anyone wants to improve support for these
> other architectures, you'll be making an egg from scratch, in the hope
> that people will start using the kernels :)

Yes, imho we're still a few years away from seeing more RISC-V
systems. In terms of ARM, I do see some u-boot packages for SBCs in
bootloaders.scm, so I assume people are using them, but I agree that
x86_64 should have by far the biggest share.

> I invite you to choose, email or IRC :)

Mail sounds good to me! (I'm rather sporadically active on IRC, so mail
usually is a better bet)

> To summarize, this work needs regular but brief attention. There's not
> much feedback from the community, so we do our best and make sure the
> basics work before pushing (reboot and connect to the internet). I'm
> eager to help grow the community of people working on this, and can help
> answer questions and give advice about things like the configs.

Thank you for this. I've seen the recent thread on making linux-libre
6.5 the default kernel[0] and will have some time at hand later on
today. I could try to prepare a patch set doing this to get more
familiar with the process, which would then need a review. WDYT?

[0]: https://lists.gnu.org/archive/html/guix-devel/2023-10/msg00027.html

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu

Re: Need people to help with kernel updates

[Leo Famulari]

On Mon, Oct 9, 2023, at 12:21, Wilko Meyer wrote:
> I've seen the recent thread on making linux-libre
> 6.5 the default kernel[0] and will have some time at hand later on
> today. I could try to prepare a patch set doing this to get more
> familiar with the process, which would then need a review. WDYT?

There's actually a patch for this already:

https://issues.guix.gnu.org/66409

But it seems to have failed the QA process:

https://qa.guix.gnu.org/issue/66409

Are you able to try applying the patch on the master branch and reporting back? I figure that double-checking that it applies is the first to figuring out what's gone wrong.

On CI, at least the 6.5.6 kernel package was built:

http://ci.guix.gnu.org/eval/831643

Re: Need people to help with kernel updates

[wolf]

[-- Attachment #1: Type: text/plain, Size: 4784 bytes --]

On 2023-10-10 07:18:23 -0400, Leo Famulari wrote:
> On Mon, Oct 9, 2023, at 12:21, Wilko Meyer wrote:
> > I've seen the recent thread on making linux-libre
> > 6.5 the default kernel[0] and will have some time at hand later on
> > today. I could try to prepare a patch set doing this to get more
> > familiar with the process, which would then need a review. WDYT?
> 
> There's actually a patch for this already:
> 
> https://issues.guix.gnu.org/66409
> 
> But it seems to have failed the QA process:
> 
> https://qa.guix.gnu.org/issue/66409
> 
> Are you able to try applying the patch on the master branch and reporting back?

The patch applies cleanly:

    $ g am ~/Downloads/66409-0.mbox
    Applying: gnu: linux-libre: Update to 6.5.6.
    $ g l -2 --oneline --no-decorate 
    a2f741df37 gnu: linux-libre: Update to 6.5.6.
    f4e8baf380 gnu: nyxt: Update to 3.9.0.

> I figure that double-checking that it applies is the first to figuring out
> what's gone wrong.

Locally it builds well, verified by running:

    $ guix shell -D guix -- ./pre-inst-env guix build linux-libre

The failure did happen in building derivation of guix-cli, and it was due to a
segmentation fault:

    [ 32/ 50] compiling...	 28.0% of 25 filesbuilder for `/gnu/store/sg8bhb53kam17997prnnrx4hyr17rqjq-guix-cli.drv' failed due to signal 11 (Segmentation fault)
    @ build-failed /gnu/store/sg8bhb53kam17997prnnrx4hyr17rqjq-guix-cli.drv - 1 builder for `/gnu/store/sg8bhb53kam17997prnnrx4hyr17rqjq-guix-cli.drv' failed due to signal 11 (Segmentation fault)
    cannot build derivation `/gnu/store/jphsmd16inbhq6mbbc4mi6kn1ri0g1hi-guix-cli-modules.drv': 1 dependencies couldn't be built
    cannot build derivation `/gnu/store/zvd7snhdmklhk0rh93g4ln07kqdj13i8-guix-system-tests.drv': 1 dependencies couldn't be built
    cannot build derivation `/gnu/store/7x6hhikczg937596kbkm139jq0q00c75-guix-ea4dc87ac-modules.drv': 1 dependencies couldn't be built
    cannot build derivation `/gnu/store/hqr1jwlkp525nlr4my051gjphbg0b04z-guix-ea4dc87ac.drv': 1 dependencies couldn't be built
    cannot build derivation `/gnu/store/vfzs38yiiqr6acwsm74rhcqvky2ayl1c-profile.drv': 1 dependencies couldn't be built
    error: load-new-guix-revision: %exception (#<&store-protocol-error message: "build of `/gnu/store/vfzs38yiiqr6acwsm74rhcqvky2ayl1c-profile.drv' failed" status: 100>)
    
    Backtrace:
              19 (primitive-load "/gnu/store/nwd4fqjp31mcch4r7cx1wnnhjccq94xj-guix-data-service-0.0.1-git.893cccf/bin/.guix-data-service-process-job-real")
    In ice-9/boot-9.scm:
      1752:10 18 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
    In guix-data-service/database.scm:
       181:14 17 (_)
    In guix-data-service/jobs/load-new-guix-revision.scm:
      2177:13 16 (_ #<pg-conn 11d0420 (connected)>)
    In guix-data-service/utils.scm:
        61:17 15 (call-with-time-logging "processing revision ea4dc87ac1c7913111d491433543f8a67bf778dc" _)
    In ice-9/boot-9.scm:
      1752:10 14 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
    In guix-data-service/database.scm:
       181:14 13 (_)
    In guix-data-service/jobs/load-new-guix-revision.scm:
      2133:24 12 (_ #<pg-conn 11e1820 (connected)>)
    In ice-9/boot-9.scm:
      1747:15 11 (with-exception-handler #<procedure 7f46ed333720 at ice-9/boot-9.scm:809:2 _> _ #:unwind? _ #:unwind-for-type _)
      1752:10 10 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
      1752:10  9 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
    In guix/store.scm:
       659:37  8 (thunk)
    In guix-data-service/jobs/load-new-guix-revision.scm:
      1724:11  7 (load-new-guix-revision #<pg-conn 11d0420 (connected)> #<store-connection 256.99 7f46ed2fc370> "1" "ea4dc87ac1c7913111d491433543f8a67bf778dc" #:skip-system-tests? _)
      1409:10  6 (channel-derivations-by-system->guix-store-item _ _)
    In guix-data-service/utils.scm:
        61:17  5 (call-with-time-logging "building the channel derivation" _)
    In guix/store.scm:
      1417:15  4 (_ #<store-connection 256.99 7f46ed2fc370> _ _)
    In ice-9/boot-9.scm:
      1685:16  3 (raise-exception _ #:continuable? _)
      1685:16  2 (raise-exception _ #:continuable? _)
      1780:13  1 (_ #<&store-protocol-error message: "build of `/gnu/store/vfzs38yiiqr6acwsm74rhcqvky2ayl1c-profile.drv' failed" status: 100>)
    In unknown file:
               0 (backtrace #<undefined>)

I do not know enough yet to debug further.

> 
> On CI, at least the 6.5.6 kernel package was built:
> 
> http://ci.guix.gnu.org/eval/831643
>

W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Need people to help with kernel updates

[Wilko Meyer]

Hi Leo,

"Leo Famulari" <leo@famulari.name> writes:

> On CI, at least the 6.5.6 kernel package was built:
>
> http://ci.guix.gnu.org/eval/831643

The current iteration of updates[0] to:

4.14.327: longterm
4.19.296: longterm
5.10.198: longterm
5.15.135: longterm
5.4.258: longterm
6.1.57: longterm
6.5.7: stable

seems to be good again at least, as in:

- it builds locally
- QA[1] looks okayish as far as I can tell (no lint warnings, build
  statuses look good on more common ISA)

even though we still seem to have partial build failures on some
architectures:

- x86_64-linux and aarch64-linux build completely fine according to QA
- without any failures or unknown status
- i686-linux has mostly succeeding builds, 5 unknown
- armhf-linux has one failing and 18 unknown

[0]: https://issues.guix.gnu.org/66455
[1]: https://qa.guix.gnu.org/issue/66455

I don't know if some of that is to be expected for less common ISAs or
if it could be beneficial to invest time in looking into e.g. the
armhf-linux build failures.

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu

Re: Need people to help with kernel updates

[Leo Famulari]

On Thu, Oct 12, 2023 at 02:15:41PM +0200, Wilko Meyer wrote:
> seems to be good again at least, as in:
> 
> - it builds locally
> - QA[1] looks okayish as far as I can tell (no lint warnings, build
>   statuses look good on more common ISA)

I agree, this already counts as "good".

I pushed your patches as 06acda9715711c406f30b3a314387002244d8792

> even though we still seem to have partial build failures on some
> architectures:
> 
> - x86_64-linux and aarch64-linux build completely fine according to QA
> - without any failures or unknown status
> - i686-linux has mostly succeeding builds, 5 unknown
> - armhf-linux has one failing and 18 unknown

Things are even better since you wrote!

The only failure is of a quasi-obsolete wireguard package on i686.
Newer kernels don't require this package to provide an out-of-tree
Wireguard driver.

I see that ppc64le is still "unknown", which in this case means that the
builds are scheduled but haven't been performed yet.

Overall, the fact that qa.guix.gnu.org is successfully building across
so many architectures means that we have a strong foundation for the
future of building kernel packages in Guix.

Thanks you for taking the initiative to write the patches and manage QA
for this latest round of updates!

I'm curious, now that we've done a round, do you have any thoughts about
the work so far?

Re: Need people to help with kernel updates

[Maxim Cournoyer]

Hi,

Luis Felipe <sirgazil@zoho.com> writes:

> El 9/10/23 a las 1:22, John Kehayias escribió:
>> Hi Leo,
>>
>>
>> On Sat, Oct 07, 2023 at 02:04 PM, Leo Famulari wrote:
>>
>>> Hello,
>>>
>>> For a few years, I've been handling updates of the linux-libre kernel by
>>> myself. Now I want some more people to help with this.
>>>
>> Just wanted to say thanks for this work that goes on mostly behind the
>> scenes; I've never had any kernel hiccups related to Guix. Let's hope
>> I didn't just jinx that, but our rollbacks and generation selection at
>> boot actually make it a lot less painful to experiment here.
>>
> Same here, thanks a lot, Leo :)

+1!

-- 
Maxim

Re: Need people to help with kernel updates

[Wilko Meyer]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:
> I pushed your patches as 06acda9715711c406f30b3a314387002244d8792

Thank you for this!

> Overall, the fact that qa.guix.gnu.org is successfully building across
> so many architectures means that we have a strong foundation for the
> future of building kernel packages in Guix.

This sounds good; good to see how many builds were actually succesful
on guixes QA!

> Thanks you for taking the initiative to write the patches and manage QA
> for this latest round of updates!
>
> I'm curious, now that we've done a round, do you have any thoughts about
> the work so far?

Thanks to the scripts you've provided and the initial explaination
you've given it was fairly easy to pick up the work of bumping the minor
kernel versions so far. I already followed new kernel releases closely
before, so figuring out when actions are required (as in when new minor
versions were available) wasn't too much of an issue. With 6.6 around
the corner soonish, I wonder how making a new kernel config for that
major release will go/what there'll be to learn in that regards. Until
now the whole process seems to be quite easy to pick up, but does
require constant recurring attention around each new minor release.

There's been a 6.1.58 stable release a few hours ago[0], with changing
mostly affecting the NFS subsystem (changes being almost exclusively
reverts of commits). I created a patch[1] for this, if you'd have a
minute to spare (or anyone else with appropriate rights to apply it on
the kernel-updates branch; as I can't do it myself), we could see if it
builds/merge it to master later on if it looks good.

[0]: https://lore.kernel.org/lkml/2023101518-subscript-entity-be71@gregkh/
[1]: https://issues.guix.gnu.org/66568

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu

Re: Need people to help with kernel updates

[Leo Famulari]

Hello everyone,

It seems that linux-libre 6.6 will be released soon.

Would anyone like to try their hand at packaging it for Guix? Wilko, do
you feel up to it?

As I mentioned previously, this requires a bit more work than minor
kernel upgrades because the kernel configs need to be updated.

I'm happy to assist with this, give advice, etc.

Leo

Re: Need people to help with kernel updates

[Wilko Meyer]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> It seems that linux-libre 6.6 will be released soon.

Yes, probably within the next(?) week if I read this mail[0] on lkml
right as there'll probably be a rc8. There already seem to be a 6.6
branch in the linux-libre project introducing the deblob scripts:

commit 8437b2928c7fd032657f571974c004130f940956 (HEAD -> scripts/6.6,
tag: scripts/v6.6-rc7-gnu, origin/scripts/6.6)

> Would anyone like to try their hand at packaging it for Guix? Wilko, do
> you feel up to it?

Sure thing! I'll start packaging it as soon as 6.6 is released and the
deblob scripts for that release are ready.

> As I mentioned previously, this requires a bit more work than minor
> kernel upgrades because the kernel configs need to be updated.
>
> I'm happy to assist with this, give advice, etc.

Thank you for this, I'll reach out if I experience any blockers or need
help during the process of packaging the 6.6 release.

[0]: https://lore.kernel.org/all/CAHk-=whqsbGgnoeYeEuP9fabaZrpPDA=SySMBe3TfQQqVMHvBA@mail.gmail.com/  

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu