From mboxrd@z Thu Jan 1 00:00:00 1970 From: Efraim Flashner Subject: Re: custom kernel config Date: Sun, 7 Apr 2019 17:36:48 +0300 Message-ID: <20190407143648.GA1337@macbook41> References: <20190401180434.GF21029@macbook41> <87wokb1pg1.fsf@gmail.com> <20190403190404.GB1203@macbook41> <871s2iuavs.fsf@mimimi.i-did-not-set--mail-host-address--so-tickle-me> <20190403202758.GC1203@macbook41> <87zhp6rwfz.fsf@gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="oC1+HKm2/end4ao3" Return-path: Received: from eggs.gnu.org ([209.51.188.92]:40135) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hD8um-0003tq-S8 for guix-devel@gnu.org; Sun, 07 Apr 2019 10:36:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hD8uk-00028r-7S for guix-devel@gnu.org; Sun, 07 Apr 2019 10:36:56 -0400 Received: from flashner.co.il ([178.62.234.194]:40792) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hD8ui-00025Y-TE for guix-devel@gnu.org; Sun, 07 Apr 2019 10:36:54 -0400 Content-Disposition: inline In-Reply-To: <87zhp6rwfz.fsf@gmail.com> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Chris Marusich Cc: guix-devel@gnu.org --oC1+HKm2/end4ao3 Content-Type: multipart/mixed; boundary="TB36FDmn/VVEgNH/" Content-Disposition: inline --TB36FDmn/VVEgNH/ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 04, 2019 at 01:44:48AM -0700, Chris Marusich wrote: > Efraim Flashner writes: >=20 > > On Wed, Apr 03, 2019 at 09:49:59PM +0200, Pierre Neidhardt wrote: > >> Thank you Efraim, this is awesome! >=20 > I second this remark! It's really helpful to see how you're doing it. > Thank you for taking the time to share with us! >=20 > >> > tar xf $(guix build linux-libre -S) > >> > cd linux-libre > >> > touch .config > >> > guix environment linux-libre > >> > make localmodconfig > >> > > >> > copy that into .config > >>=20 > >> Copy what into .config? > > > > the .config you just created at the root of the source tarball >=20 > I think there is some confusion here. When I run "touch .config" and > then run "make localmodconfig", the .config file is not modified. > Perhaps you meant that we need to take the stdout from "make > localmodconfig", save it, and copy it (after massaging it) back into the > .config file? Yeah, basically >=20 > > running make localconfig spits out a bunch of pseudo-error messages > > about modules that are missing in .config. If you copy that whole > > message, minus the lines starting with 'WARING', into .config and > > then format it as UPERCASE_OPTION=3D(y|m) it gives the kernel a config > > that is usable for it. > > > > so turning lines that look like: > > module nouveau did not have configs CONFIG_DRM_NOUVEAU > > > > into lines that look like: > > CONFIG_DRM_NOUVEAU=3Dm >=20 > I read up a little on the various targets. The documentation you > mentioned was helpful, "make help" was helpful, the Makefile source was > somewhat helpful, and the contents of > ./scripts/kconfig/streamline_config.pl were helpful for understanding > the intent of the "localmodconfig" target. >=20 > It sounds like the intended use of "localmodconfig" target is to modify > an existing, non-empty .config file. With that in mind, how does the > procedure you describe compare to something like the following? >=20 > 1) Invoke "guix repl". Then run this to determine the config that would > normally be used to configure the default linux-libre kernel: >=20 > scheme@(guix-user)> (assoc "kconfig" (package-native-inputs linux-libre= )) > $1 =3D ("kconfig" "/gnu/store/cwghfr06cadj2ss1ya6whgczpcba58z3-guix-mod= ule-union/share/guile/site/2.2/gnu/packages/aux-files/linux-libre/5.0-x86_6= 4.conf") >=20 > 2) Copy that file out of the store into .config. >=20 > 3) Run "make olddefconfig" to non-interactively update the .config file. >=20 > 4) Run "make localmodconfig" to streamline the .config based on what's > currently loaded. I expected this target to also be non-interactive, > but to my surprise it still asked me to choose what to set for a few > options. >=20 > If you do it this way, you don't have to fix up formatting. However, > maybe it doesn't produce the same kind of configuration you had in mind. > What do you think? >=20 This doesn't create a config that is as small as possible, which is what I was looking for. > > Currently this doesn't take into account the initrd. >=20 > Do you mean that there are modules or configuration which the initrd > might need, which this procedure might accidentally disable? >=20 Yeah. I get different errors about missing ahci.ko or scsi.ko, so it's going to take a bit more work. > --=20 > Chris I've started writing up a blog post about it (which I've attached), but it doesn't work as-is. More work incoming later I guess. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --TB36FDmn/VVEgNH/ Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="custom-kernel.md" Content-Transfer-Encoding: quoted-printable title: Creating and using a custom Linux kernel on Guix System date: 2019-04-04 00:00 author: Efraim Flashner tags: kernel, customization --- Guix is, at its core, a source based distribution with substitutes, and as = such building packages from their source code is an expected part of regula= r package installations and upgrades. Given this starting point, it makes = sense that efforts are made to reduce the amount of time spent compiling pa= ckages, and recent changes and upgrades to the building and distribution of= substitutes continues to be a topic of discussion within Guix. One of the= packages which I prefer to not build myself is the Linux-Libre kernel. Th= e kernel, while not requiring an overabundance of RAM to build, does take a= very long time on my build machine (which my children argue is actually th= eir Kodi computer), and I will often delay reconfiguring my laptop while I = want for a substitute to be prepared by the official build farm. The offic= ial kernel configuration, as is the case with many GNU/Linux distributions,= errs on the side of inclusiveness, and this is really what causes the buil= d to take such a long time when I build the package for myself. The Linux kernel, however, can also just be described as a package installe= d on my machine, and as such can be customized just like any other package.= The procedure is a little bit different, although this is primarily due t= o the nature of how the package defintion is written. The linux-libre kernel package definition is actually a procedure which cre= ates a package. ```scheme (define* (make-linux-libre version hash supported-systems #:key ;; A function that takes an arch and a variant. ;; See kernel-config for an example. (extra-version #f) (configuration-file #f) (defconfig "defconfig") (extra-options %default-extra-linux-options) (patches (list %boot-logo-patch))) ...) ``` The current linux-libre package is for the 5.0.x series, and is declared li= ke this: ```scheme (define-public linux-libre (make-linux-libre %linux-libre-version %linux-libre-hash '("x86_64-linux" "i686-linux" "armhf-linux" "aarch64-li= nux") #:patches %linux-libre-5.0-patches #:configuration-file kernel-config)) ``` Any keys which are not assigned values inherit their default value from the= make-linux-libre definition. When comparing the two snippets above, you m= ay notice that the code comment in the first doesn't actually refer to the = extra-version keyword; it is actually for configuration-file. Because of t= his, it is not actually easy to include a custom kernel configuration from = the definition, but don't worry, there are other ways to work with what we = do have. There are two ways to create a kernel with a custom kernel configuration. = The first is to provide a standard '.config' file during the build process = by including an actual '.config' file as a native-input to our custom kerne= l. The following is a snippet from the custom 'configure phase of the make= -linux-libre package definition: ```scheme (let ((build (assoc-ref %standard-phases 'build)) (config (assoc-ref (or native-inputs inputs) "kconfig"))) ;; Use a custom kernel configuration file or a default ;; configuration file. (if config (begin (copy-file config ".config") (chmod ".config" #o666)) (invoke "make" ,defconfig)) ``` Below is a sample kernel package for one of my computers: ```scheme (define-public linux-libre-E2140 (let ((base ((@@ (gnu packages linux) make-linux-libre) (@@ (gnu packages linux) %linux-libre-version) (@@ (gnu packages linux) %linux-libre-hash) '("x86_64-linux") #:extra-version "E2140" #:patches (@@ (gnu packages linux) %linux-libre-5.0-patches)))) (package (inherit base) (native-inputs `(("kconfig" ,(local-file "E2140.config")) ,@(package-native-inputs base)))))) ``` In the same directory as the file defining linux-libre-E2140 is a file name= d E2140.config, which is an actual kernel configuration file. I left the d= efconfig keyword of make-linux-libre blank, so the only kernel configuratio= n in the package is the one which I included as a native-input. The second way to create a custom kernel is to pass a new value to the extr= a-options keyword of the make-linux-libre procedure. The extra-options key= word works with another function defined right below it: ```scheme (define %default-extra-linux-options `(;; https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00039.html ("CONFIG_DEVPTS_MULTIPLE_INSTANCES" . #t) ;; Modules required for initrd: ("CONFIG_NET_9P" . m) ("CONFIG_NET_9P_VIRTIO" . m) ("CONFIG_VIRTIO_BLK" . m) ("CONFIG_VIRTIO_NET" . m) ("CONFIG_VIRTIO_PCI" . m) ("CONFIG_VIRTIO_BALLOON" . m) ("CONFIG_VIRTIO_MMIO" . m) ("CONFIG_FUSE_FS" . m) ("CONFIG_CIFS" . m) ("CONFIG_9P_FS" . m))) (define (config->string options) (string-join (map (match-lambda ((option . 'm) (string-append option "=3Dm")) ((option . #t) (string-append option "=3Dy")) ((option . #f) (string-append option "=3Dn"))) options) "\n")) ``` And in the custom configure script from the make-linux-libre package: ```scheme ;; Appending works even when the option wasn't in the ;; file. The last one prevails if duplicated. (let ((port (open-file ".config" "a")) (extra-configuration ,(config->string extra-options))) (display extra-configuration port) (close-port port)) (invoke "make" "oldconfig")))) ``` So by not providing a configuraion-file the '.config' starts blank, and the= n we write into it the collection of flags that we want. Here's another cu= stom kernel which I have: ```scheme (define %macbook41-full-config (append %macbook41-config-options %filesystems %efi-support %emulation (@@ (gnu packages linux) %default-extra-linux-options))) (define-public linux-libre-macbook41 ((@@ (gnu packages linux) make-linux-libre) (@@ (gnu packages linux) %lin= ux-libre-version) (@@ (gnu packages linux) %linux-libre-hash) '("x86_64-linux") #:extra-version "macbook41" #:patches (@@ (gnu packages linux) %linux-libre-5.0-p= atches) #:extra-options %macbook41-config-options)) ``` =46rom the above example %filesystems is a collection of flags enabling dif= ferent filesystem support, %efi-support enables EFI support and %emulation = enables my x86_64-linux machine to act in 32-bit mode also. %defalt-extra= -linux-options are the ones quoted above, which had to be added in since I = used replaced them in the extra-options keyword. This all sounds like it should be doable, but how does one even know which = modules are required for their system? The two places I found most helpful= to try to answer this question were the [Gentoo Handbook](https://wiki.gen= too.org/wiki/Handbook:AMD64/Installation/Kernel), and the [documentation](h= ttps://www.kernel.org/doc/html/latest/admin-guide/README.html?highlight=3Dl= ocalmodconfig) from the kernel itself. From the kernel documentation, it s= eems that "make localmodconfig" is the command we want. In order to actually run `make localmodconfig` we first need to get and unp= ack the kernel source code: ```shell tar xf $(guix build linux-libre --source) ``` Once inside the directory containing the source code run `touch .config` to= create an initial, empty '.config' to start with. `make localmodconfig` w= orks by seeing what you already have in '.config' and letting you know what= you're missing. If the file is blank then you're missing everything. The= next step is to run ```shell make localmodconfig ``` and note the output. Do note that the '.config' file is still empty. The = output generally contains two types of warnings. The first start with "WAR= NING" and can actually be ignored in our case. The second read: ```shell module pcspkr did not have configs CONFIG_INPUT_PCSPKR ``` For each of these lines, copy the CONFIG_XXXX_XXXX portion into the '.confi= g' in the directory, and append "=3Dm", so in the end it looks like this: ```shell CONFIG_INPUT_PCSPKR=3Dm CONFIG_VIRTIO=3Dm ``` After copying all the configuration options, run `make localmodconfig` agai= n to make sure that you don't have any output starting with "module". Afte= r all of these machine specific modules there are a couple more left that a= re also needed. CONFIG_MODULES is necessary so that you can build and load= modules separately and not have everything built into the kernel. CONFIG_= BLK_DEV_SD is required for reading from hard drives. It is possible that t= here are other modules which you will need. The second way to setup the kernel configuration makes more use of Guix's f= eatures and allows you to share configuration segments between different ke= rnels. For example, all machines using EFI to boot have a number of EFI co= nfiguration flags that they need. It is likely that all the kernels will s= hare a list of filesystems to support. #### About GNU Guix [GNU=C2=A0Guix](https://www.gnu.org/software/guix) is a transactional packa= ge manager and an advanced distribution of the GNU system that [respects user freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.ht= ml). Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines. In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through [Guile](https://www.gnu.org/software/guile) programming interfaces and extensions to the [Scheme](http://schemers.org) language. --TB36FDmn/VVEgNH/-- --oC1+HKm2/end4ao3 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlyqCv0ACgkQQarn3Mo9 g1EgZQ/+KnEF9Ban86zp+Qj2j6ux8Pw1prg7qYQBOcq/9GTSkWT2CqehWsZVLxLT aUFqjxXLhLKLZZMAZNxWkPpwqM0ntrVu/42un3nBIUugFac+OXCoSkakrnW7QFlF ABHwRkF5mm9lvAipCQ5UuR/6jpn/zvtdnLDy5InmXC9RqqfakIjI6fPRAHEGPYJA CBxckm4dROSg5n9VRuZ9At3C/6asUajpLLcJvk+qzCW+DeOPStd4xWa5NYtggQu2 gjGQs230o0myW+scSX6/MB4ineZrn9qN3OpUmut2YpKUH/RRvyScDm8Ba0AnmmgL 1eSjXnfMMdv8frRlB7IVRakCJCUBYp0+3+YQiwWVByh5ON8Z4F8LsX3fiEehqI7Z YSkPGRZ0X6sb5qTC6xhGNM+2Kd5jlVeb30sa6uLPzOdnl7hPqtJuSeo7MDHnpm8e ccCmKcR8PemSU4U1x/h1qwqy99NF+rE48bjFVXGE+gs5N7QTfLqhJxfmtYj5kxju 5UHaSVveliD7zvP2afnGI0IvA2sHHka1YPGfgfUBZ5IGDLilUe65BTHSaskNmNLV wPKWS4Z8ZCmgAXaC8uTIF3xNC2XA9B1mucOTKdw5rxfAT3Uvg1+Y4Wc8HWAAwVCu zkLi68x1PC/M5bsdc3OH7ARuRRUxkq/VbBYz492KZyyO+t4OByY= =wGvc -----END PGP SIGNATURE----- --oC1+HKm2/end4ao3--