all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Kernel modules in initrd
@ 2018-02-22 21:17 Andreas Enge
  2018-02-22 21:29 ` Jan Nieuwenhuizen
                   ` (2 more replies)
  0 siblings, 3 replies; 21+ messages in thread
From: Andreas Enge @ 2018-02-22 21:17 UTC (permalink / raw)
  To: guix-devel

Hello,

recently I had an unpleasant experience in installing GuixSD: After booting
with the USB key and following the installation instructions, then rebooting
into the installed system, the "/" file system was not found: neither using
file system labels, nor device nodes.

The problem turned out to be that the disk of the machine needed special
kernel modules, and adding
  (initrd (lambda (file-systems . rest)
            (apply base-initrd file-systems
                   #:extra-modules '("mptbase" "mptsas" "mptscsih")
                                   rest)))
to the operating-system declaration solved the problem.
However, it took us quite some time and several trials to diagnose the
problem in the first place.

So I wonder:
1) Could we add more kernel modules to the base-initrd, whenever people
   report that new ones are needed? For instance, the berlin server has
   modules (list "megaraid_sas" "libsas" "scsi_transport_sas").
2) Better yet, since we managed to boot on the USB key and see the disk:
   Why not have the same modules in the initrd of the installation image
   and of the installed default system? Not being able to see the disk
   in the beginning would at least have made the process fail early on,
   and moreover there seem to be more modules on the USB key. And what
   fits onto a 1 GB USB key should also easily fit on a hard disk...

What do you think?

Andreas

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 21:17 Kernel modules in initrd Andreas Enge
@ 2018-02-22 21:29 ` Jan Nieuwenhuizen
  2018-02-22 21:44   ` Danny Milosavljevic
  2018-02-22 21:34 ` Danny Milosavljevic
  2018-02-22 22:01 ` Mark H Weaver
  2 siblings, 1 reply; 21+ messages in thread
From: Jan Nieuwenhuizen @ 2018-02-22 21:29 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Andreas Enge writes:

> The problem turned out to be that the disk of the machine needed special
> kernel modules, and adding
>   (initrd (lambda (file-systems . rest)
>             (apply base-initrd file-systems
>                    #:extra-modules '("mptbase" "mptsas" "mptscsih")
>                                    rest)))
> to the operating-system declaration solved the problem.

Sounds familiar, when installing GuixSD on older hardware, I needed these

    (initrd (lambda (file-systems . rest)
              (apply base-initrd file-systems
                     #:extra-modules '("pata_via" "pata_acpi" "sata_via")
                     rest)))

> However, it took us quite some time and several trials to diagnose the
> problem in the first place.

Yes.

> So I wonder:
> 1) Could we add more kernel modules to the base-initrd, whenever people
>    report that new ones are needed? For instance, the berlin server has
>    modules (list "megaraid_sas" "libsas" "scsi_transport_sas").
> 2) Better yet, since we managed to boot on the USB key and see the disk:
>    Why not have the same modules in the initrd of the installation image
>    and of the installed default system? Not being able to see the disk
>    in the beginning would at least have made the process fail early on,
>    and moreover there seem to be more modules on the USB key. And what
>    fits onto a 1 GB USB key should also easily fit on a hard disk...
>
> What do you think?

I'm not experienced in this, otoh if things "just work", that's always
nice.
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 21:17 Kernel modules in initrd Andreas Enge
  2018-02-22 21:29 ` Jan Nieuwenhuizen
@ 2018-02-22 21:34 ` Danny Milosavljevic
  2018-02-22 21:50   ` [PATCH] linux-initrd: Add ATA and SAS modules to the default set of modules Danny Milosavljevic
  2018-02-22 21:53   ` Kernel modules in initrd Andreas Enge
  2018-02-22 22:01 ` Mark H Weaver
  2 siblings, 2 replies; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-22 21:34 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Hi Andreas,

On Thu, 22 Feb 2018 22:17:07 +0100
Andreas Enge <andreas@enge.fr> wrote:

> The problem turned out to be that the disk of the machine needed special
> kernel modules, and adding
>   (initrd (lambda (file-systems . rest)
>             (apply base-initrd file-systems
>                    #:extra-modules '("mptbase" "mptsas" "mptscsih")
>                                    rest)))
> to the operating-system declaration solved the problem.

> So I wonder:
> 1) Could we add more kernel modules to the base-initrd, whenever people
>    report that new ones are needed? For instance, the berlin server has
>    modules (list "megaraid_sas" "libsas" "scsi_transport_sas").

Sure.

> 2) Better yet, since we managed to boot on the USB key and see the disk:
>    Why not have the same modules in the initrd of the installation image
>    and of the installed default system? 

I don't think that that is what happened.

The initrd is a very early root filesystem which is supposed to contain
enough stuff (programs and data) to be able to mount the actual root
filesystem.

When you booted from the USB key the system DIDN'T need mptsas to find
ITS root filesystem.  So it booted fine and in the end of it all you had
a working GNU Linux, with udev.

Then udev saw the hard disk and udev went and loaded the kernel
module (from the root filesystem, not from the initrd), as it was
supposed to do.

It's unfortunate that the initrd is needed, but let's just add all
maybe-required kernel modules to it.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 21:29 ` Jan Nieuwenhuizen
@ 2018-02-22 21:44   ` Danny Milosavljevic
  0 siblings, 0 replies; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-22 21:44 UTC (permalink / raw)
  To: Jan Nieuwenhuizen; +Cc: guix-devel

Hi,

On Thu, 22 Feb 2018 22:29:25 +0100
Jan Nieuwenhuizen <janneke@gnu.org> wrote:

>     (initrd (lambda (file-systems . rest)
>               (apply base-initrd file-systems
>                      #:extra-modules '("pata_via" "pata_acpi" "sata_via")
>                      rest)))

Hmm, see gnu/system/linux-initrd.scm linux-modules .

It should already have pata_acpi, pata_atiixp.  We probably should add pata_via, sata_via there...

> >    modules (list "megaraid_sas" "libsas" "scsi_transport_sas").

Sure, also to linux-modules.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* [PATCH] linux-initrd: Add ATA and SAS modules to the default set of modules.
  2018-02-22 21:34 ` Danny Milosavljevic
@ 2018-02-22 21:50   ` Danny Milosavljevic
  2018-02-27 15:03     ` Ludovic Courtès
  2018-02-22 21:53   ` Kernel modules in initrd Andreas Enge
  1 sibling, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-22 21:50 UTC (permalink / raw)
  To: guix-devel

* gnu/system/linux-initrd.scm (base-initrd)[linux-modules]: Add "pata_via",
"sata_via", "megaraid_sas", "libsas", "scsi_transport_sas", "mptbase",
"mptsas", "mptscsih".
---
 gnu/system/linux-initrd.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 330438bce..7170d1c0e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -292,8 +292,9 @@ loaded at boot time in the order in which they appear."
       "dm-crypt" "xts" "serpent_generic" "wp512" ;for encrypted root partitions
       "nls_iso8859-1"                            ;for `mkfs.fat`, et.al
       ,@(if (string-match "^(x86_64|i[3-6]86)-" (%current-system))
-            '("pata_acpi" "pata_atiixp"    ;for ATA controllers
-              "isci")                      ;for SAS controllers like Intel C602
+            '("pata_acpi" "pata_atiixp" "pata_via" "sata_via"  ;for ATA controllers
+              "isci" "megaraid_sas" "libsas" "scsi_transport_sas" ;for SAS controllers
+              "mptbase" "mptsas" "mptscsih") ;for SAS controllers
             '())
       ,@(if (or virtio? qemu-networking?)
             virtio-modules

^ permalink raw reply related	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 21:34 ` Danny Milosavljevic
  2018-02-22 21:50   ` [PATCH] linux-initrd: Add ATA and SAS modules to the default set of modules Danny Milosavljevic
@ 2018-02-22 21:53   ` Andreas Enge
  1 sibling, 0 replies; 21+ messages in thread
From: Andreas Enge @ 2018-02-22 21:53 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Hello Danny,

On Thu, Feb 22, 2018 at 10:34:03PM +0100, Danny Milosavljevic wrote:
> When you booted from the USB key the system DIDN'T need mptsas to find
> ITS root filesystem.  So it booted fine and in the end of it all you had
> a working GNU Linux, with udev.
> 
> Then udev saw the hard disk and udev went and loaded the kernel
> module (from the root filesystem, not from the initrd), as it was
> supposed to do.

thanks for the explanations, I agree with your analysis.
So the patch you propose looks good to me.

Andreas

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 21:17 Kernel modules in initrd Andreas Enge
  2018-02-22 21:29 ` Jan Nieuwenhuizen
  2018-02-22 21:34 ` Danny Milosavljevic
@ 2018-02-22 22:01 ` Mark H Weaver
  2018-02-23  1:00   ` Danny Milosavljevic
  2018-02-23 22:39   ` Ludovic Courtès
  2 siblings, 2 replies; 21+ messages in thread
From: Mark H Weaver @ 2018-02-22 22:01 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> recently I had an unpleasant experience in installing GuixSD: After booting
> with the USB key and following the installation instructions, then rebooting
> into the installed system, the "/" file system was not found: neither using
> file system labels, nor device nodes.
>
> The problem turned out to be that the disk of the machine needed special
> kernel modules, and adding
>   (initrd (lambda (file-systems . rest)
>             (apply base-initrd file-systems
>                    #:extra-modules '("mptbase" "mptsas" "mptscsih")
>                                    rest)))
> to the operating-system declaration solved the problem.
> However, it took us quite some time and several trials to diagnose the
> problem in the first place.
>
> So I wonder:
> 1) Could we add more kernel modules to the base-initrd, whenever people
>    report that new ones are needed? For instance, the berlin server has
>    modules (list "megaraid_sas" "libsas" "scsi_transport_sas").

We need a better system for dealing with this.

I don't think that loading every kernel module that any GuixSD user ever
needed during the early boot process is a good approach.  To make
matters worse, it's easy for a user to add modules to their initrd, but
there's no easy way to remove modules from the default set.

I don't want *every* GuixSD user to unconditionally load, into their
kernels during early boot, code to support LSI MegaRAID SAS cards, just
because there exists one GuixSD user out there who needs it.

Every extra loaded kernel module means more RAM usage in the kernel, a
larger initrd image that must be loaded by possibly slow bootloaders,
and code complexity in the running kernel, leading to a greater attack
surface for possible security exploits.  For example, last year some
memory corruption bugs were found in the LSI MegaRAID SAS module.  See
<https://patchwork.kernel.org/patch/9585337/>.

To make this easier, I think the right approach is to include many
modules like these to our installation image initrd, and then to
automatically detect which modules are needed for booting.  A future
easy installer could automatically add those modules to the OS config,
but in the meantime we could simply print a message recommending that
the user should add the needed modules to their initrd config.

What do you think?

      Mark

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 22:01 ` Mark H Weaver
@ 2018-02-23  1:00   ` Danny Milosavljevic
  2018-02-23 14:28     ` Danny Milosavljevic
  2018-02-23 22:39   ` Ludovic Courtès
  1 sibling, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-23  1:00 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guix-devel

Hi Mark,

On Thu, 22 Feb 2018 17:01:11 -0500
Mark H Weaver <mhw@netris.org> wrote:

> Every extra loaded kernel module means more RAM usage in the kernel, a
> larger initrd image that must be loaded by possibly slow bootloaders,
> and code complexity in the running kernel, leading to a greater attack
> surface for possible security exploits.  For example, last year some
> memory corruption bugs were found in the LSI MegaRAID SAS module.  See
> <https://patchwork.kernel.org/patch/9585337/>.

Good points.

> To make this easier, I think the right approach is to include many
> modules like these to our installation image initrd, and then to
> automatically detect which modules are needed for booting.  A future
> easy installer could automatically add those modules to the OS config,
> but in the meantime we could simply print a message recommending that
> the user should add the needed modules to their initrd config.

Good idea.

Another possibility that maybe would make the current situation less bad
would be to put udev-static into the initrd [1] and basically make it only
load the required modules on demand.

Also, udev uses a lot of tools like grep etc - we could use busybox to
get the size of the initrd down again then.

I definitely agree that we should only add modules possibly required for
early booting (until the rootfs is mounted) and not all the modules - that
would be insanely big.

On another note, let's please make the error detection and reporting better.

It should be easy to find unclaimed nodes by scanning /sys/class/pci_bus or
walking /sys/devices/pci0000:00, 
trying to find whether each has a "driver" entry and that would have caught
this problem and improved the diagnostics a lot.  This is not 1990 where
when we had no driver we didn't know the hardware was there.  We do know now.

[1] https://www.redhat.com/archives/fedora-devel-list/2004-May/msg01008.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-23  1:00   ` Danny Milosavljevic
@ 2018-02-23 14:28     ` Danny Milosavljevic
  2018-02-23 23:02       ` Andreas Enge
  0 siblings, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-23 14:28 UTC (permalink / raw)
  To: Mark H Weaver, guix-devel

No, wait, according to https://unix.stackexchange.com/questions/43699/debian-does-not-detect-serial-pci-card-after-reboot/43723#43723 ,
the kernel should be doing that even without udev.  Are we sure we need to manually modprobe the stuff in gnu/build/linux-boot.scm in
the first place?  I think we should just add kmod to the initrd - that's it.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-22 22:01 ` Mark H Weaver
  2018-02-23  1:00   ` Danny Milosavljevic
@ 2018-02-23 22:39   ` Ludovic Courtès
  2018-02-24  8:28     ` ng0
  1 sibling, 1 reply; 21+ messages in thread
From: Ludovic Courtès @ 2018-02-23 22:39 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guix-devel

Hello,

Mark H Weaver <mhw@netris.org> skribis:

> To make this easier, I think the right approach is to include many
> modules like these to our installation image initrd, and then to
> automatically detect which modules are needed for booting.  A future
> easy installer could automatically add those modules to the OS config,
> but in the meantime we could simply print a message recommending that
> the user should add the needed modules to their initrd config.

FWIW I have code that I was planning to polish and submit (hopefully
next week) that determines modules needed for a particular device by
walking /sys.  In addition, I thought we could make the initrd module
list a first class field of ‘operating-system’.  With these two pieces,
‘guix system’ should be able to warn when a module is missing.

Ludo’.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-23 14:28     ` Danny Milosavljevic
@ 2018-02-23 23:02       ` Andreas Enge
  2018-02-25 11:43         ` Danny Milosavljevic
  0 siblings, 1 reply; 21+ messages in thread
From: Andreas Enge @ 2018-02-23 23:02 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Hello,

On Fri, Feb 23, 2018 at 03:28:55PM +0100, Danny Milosavljevic wrote:
> No, wait, according to https://unix.stackexchange.com/questions/43699/debian-does-not-detect-serial-pci-card-after-reboot/43723#43723 ,
> the kernel should be doing that even without udev.  Are we sure we need to manually modprobe the stuff in gnu/build/linux-boot.scm in
> the first place?  I think we should just add kmod to the initrd - that's it.

in that case, if I understand correctly, the security question would not
be a problem any more, right, as only really needed modules would be loaded
by the kernel? Then we could add more modules to the initrd.

Andreas

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-23 22:39   ` Ludovic Courtès
@ 2018-02-24  8:28     ` ng0
  2018-02-27 15:04       ` Ludovic Courtès
  0 siblings, 1 reply; 21+ messages in thread
From: ng0 @ 2018-02-24  8:28 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

Ludovic Courtès transcribed 0.8K bytes:
> Hello,
> 
> Mark H Weaver <mhw@netris.org> skribis:
> 
> > To make this easier, I think the right approach is to include many
> > modules like these to our installation image initrd, and then to
> > automatically detect which modules are needed for booting.  A future
> > easy installer could automatically add those modules to the OS config,
> > but in the meantime we could simply print a message recommending that
> > the user should add the needed modules to their initrd config.
> 
> FWIW I have code that I was planning to polish and submit (hopefully
> next week) that determines modules needed for a particular device by
> walking /sys.  In addition, I thought we could make the initrd module
> list a first class field of ‘operating-system’.  With these two pieces,
> ‘guix system’ should be able to warn when a module is missing.
> 
> Ludo’.

I'm willing to test those, it's probably less time consuming than figuring out
the corretc modules the Hetzner CLoud instance needs.
-- 
ng0 ::  https://n0.is | https://crash.cx
A88C8ADD129828D7EAC02E52E22F9BBFEE348588

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-23 23:02       ` Andreas Enge
@ 2018-02-25 11:43         ` Danny Milosavljevic
  2018-02-26 15:20           ` Ludovic Courtès
  0 siblings, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-25 11:43 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Hi Andreas,

On Sat, 24 Feb 2018 00:02:39 +0100
Andreas Enge <andreas@enge.fr> wrote:

> On Fri, Feb 23, 2018 at 03:28:55PM +0100, Danny Milosavljevic wrote:
> > No, wait, according to https://unix.stackexchange.com/questions/43699/debian-does-not-detect-serial-pci-card-after-reboot/43723#43723 ,
> > the kernel should be doing that even without udev.  Are we sure we need to manually modprobe the stuff in gnu/build/linux-boot.scm in
> > the first place?  I think we should just add kmod to the initrd - that's it.  
> 
> in that case, if I understand correctly, the security question would not
> be a problem any more, right, as only really needed modules would be loaded
> by the kernel? Then we could add more modules to the initrd.

Yes.

I've got it to work now.  I've got a very minimal static kmod into the initrd and
that's now only loading modules for which supported hardware is present.

On the other hand, the initrd got 800 kiB larger - I'm not sure why modprobe
is so big... hmm...

Let's see the part Ludo adds to guix system init.  Maybe we can also replace
modprobe by it - if we want.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-25 11:43         ` Danny Milosavljevic
@ 2018-02-26 15:20           ` Ludovic Courtès
  2018-02-26 16:26             ` Danny Milosavljevic
  0 siblings, 1 reply; 21+ messages in thread
From: Ludovic Courtès @ 2018-02-26 15:20 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Hello,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> I've got it to work now.  I've got a very minimal static kmod into the initrd and
> that's now only loading modules for which supported hardware is present.
>
> On the other hand, the initrd got 800 kiB larger - I'm not sure why modprobe
> is so big... hmm...

Yeah.  If possible, I’d like to keep the initrd pure Scheme, to keep it
smallish.

I suppose the bits I’ve promised to detect necessary modules based on
modules.alias and /sys should be useful here?

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-26 15:20           ` Ludovic Courtès
@ 2018-02-26 16:26             ` Danny Milosavljevic
  2018-02-27 15:02               ` Ludovic Courtès
  0 siblings, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-26 16:26 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

Hi Ludo,

On Mon, 26 Feb 2018 16:20:08 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> I suppose the bits I’ve promised to detect necessary modules based on
> modules.alias and /sys should be useful here?

Definitely!

I've posted a working patch series (v3, "Load Linux module only when supported
hardware is present") of the current kmod-based autoloader to guix-patches.
It's using an old-ish kmod - but otherwise it works fine.

So what we would need next is something like modprobe written in guile.

I think that ./gnu/build/linux-modules.scm load-linux-module* already does that.

The only part missing is to replace %modprobe-wrapper by a guile script
which imports ./gnu/build/linux-modules.scm load-linux-module*, interprets
some of the modprobe options and then masquerades as modprobe without invoking
modprobe.

The reason is that the kernel calls us back when it needs a module (for
example when you mount something and it needs a charset converter or something).

(the name of the executable invoked is configured via /proc/sys/kernel/modprobe)

This can happen quite early in the boot process - and it took me some time
to get the order and also the environment variables right.  We should be able
to reuse that for the pure Guile modprobe.

After the patch we don't *manually* load any of the modules - and all the
tests work.  I booted my system with it - that also works.

P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked it?
glibc is like 50000 kiB.  In that case saving 800 kiB is not really worth it...

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-26 16:26             ` Danny Milosavljevic
@ 2018-02-27 15:02               ` Ludovic Courtès
  2018-02-27 19:32                 ` Danny Milosavljevic
  0 siblings, 1 reply; 21+ messages in thread
From: Ludovic Courtès @ 2018-02-27 15:02 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> So what we would need next is something like modprobe written in guile.
>
> I think that ./gnu/build/linux-modules.scm load-linux-module* already does that.
>
> The only part missing is to replace %modprobe-wrapper by a guile script
> which imports ./gnu/build/linux-modules.scm load-linux-module*, interprets
> some of the modprobe options and then masquerades as modprobe without invoking
> modprobe.
>
> The reason is that the kernel calls us back when it needs a module (for
> example when you mount something and it needs a charset converter or something).
>
> (the name of the executable invoked is configured via /proc/sys/kernel/modprobe)
>
> This can happen quite early in the boot process - and it took me some time
> to get the order and also the environment variables right.  We should be able
> to reuse that for the pure Guile modprobe.
>
> After the patch we don't *manually* load any of the modules - and all the
> tests work.  I booted my system with it - that also works.

I see, that’s a workable plan, we should do that!

> P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked it?
> glibc is like 50000 kiB.  In that case saving 800 kiB is not really worth it...

One of the packages that ends up in the initrd must be dynamically
linked.  You need to find out which one it is.

Thanks for the explanations!

Ludo’.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [PATCH] linux-initrd: Add ATA and SAS modules to the default set of modules.
  2018-02-22 21:50   ` [PATCH] linux-initrd: Add ATA and SAS modules to the default set of modules Danny Milosavljevic
@ 2018-02-27 15:03     ` Ludovic Courtès
  0 siblings, 0 replies; 21+ messages in thread
From: Ludovic Courtès @ 2018-02-27 15:03 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> * gnu/system/linux-initrd.scm (base-initrd)[linux-modules]: Add "pata_via",
> "sata_via", "megaraid_sas", "libsas", "scsi_transport_sas", "mptbase",
> "mptsas", "mptscsih".

I think we can do that, but only once we have the code that loads
modules on demand, and assuming the initrd size remains reasonable.

Ludo’.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-24  8:28     ` ng0
@ 2018-02-27 15:04       ` Ludovic Courtès
  0 siblings, 0 replies; 21+ messages in thread
From: Ludovic Courtès @ 2018-02-27 15:04 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: guix-devel

Hello,

ng0 <ng0@crash.cx> skribis:

> Ludovic Courtès transcribed 0.8K bytes:
>> Hello,
>> 
>> Mark H Weaver <mhw@netris.org> skribis:
>> 
>> > To make this easier, I think the right approach is to include many
>> > modules like these to our installation image initrd, and then to
>> > automatically detect which modules are needed for booting.  A future
>> > easy installer could automatically add those modules to the OS config,
>> > but in the meantime we could simply print a message recommending that
>> > the user should add the needed modules to their initrd config.
>> 
>> FWIW I have code that I was planning to polish and submit (hopefully
>> next week) that determines modules needed for a particular device by
>> walking /sys.  In addition, I thought we could make the initrd module
>> list a first class field of ‘operating-system’.  With these two pieces,
>> ‘guix system’ should be able to warn when a module is missing.
>> 
>> Ludo’.
>
> I'm willing to test those, it's probably less time consuming than figuring out
> the corretc modules the Hetzner CLoud instance needs.

I’ve posted the patch series here:

  https://bugs.gnu.org/30629

Feedback welcome!

Ludo’.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-27 15:02               ` Ludovic Courtès
@ 2018-02-27 19:32                 ` Danny Milosavljevic
  2018-02-27 20:52                   ` Danny Milosavljevic
  0 siblings, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-27 19:32 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

> > P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked it?
> > glibc is like 50000 kiB.  In that case saving 800 kiB is not really worth it...  
> 
> One of the packages that ends up in the initrd must be dynamically
> linked.  You need to find out which one it is.

It's because of bash-minimal.

$ grep -r 'bash-minimal' .
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/bin/guild:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/bin/guile-config:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/bin/guile-snarf:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh
Binary file ./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/lib/guile/2.2/ccache/ice-9/popen.go matches
./gnu/store/38553wfz0jwlgbw13pk99xl79pbfx58d-guile-2.2.3/share/guile/2.2/ice-9/popen.scm:  (open-pipe* mode "/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" command))
Binary file ./gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash matches
./gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bashbug:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/sh -
./gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bashbug:CFLAGS=" -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu' -DCONF_VENDOR='unknown' -DLOCALEDIR='/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -g -O2 -Wno-parentheses -Wno-format-security"
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzdiff:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzgrep:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzless:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
./gnu/store/kgaf671a9a76k0ql1pwwjxjbj80x22mj-xz-5.2.3/bin/xzmore:#!/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash
Binary file ./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/lib/guile/2.2/ccache/ice-9/popen.go matches
./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/share/guile/2.2/ice-9/popen.scm:  (open-pipe* mode "/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" command))

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-27 19:32                 ` Danny Milosavljevic
@ 2018-02-27 20:52                   ` Danny Milosavljevic
  2018-02-28 21:49                     ` Ludovic Courtès
  0 siblings, 1 reply; 21+ messages in thread
From: Danny Milosavljevic @ 2018-02-27 20:52 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Tue, 27 Feb 2018 20:32:49 +0100
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> > > P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked it?
> > > glibc is like 50000 kiB.  In that case saving 800 kiB is not really worth it...    
> > 
> > One of the packages that ends up in the initrd must be dynamically
> > linked.  You need to find out which one it is.  
> 
> It's because of bash-minimal.

And that's because of:

./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/share/guile/2.2/ice-9/popen.scm:  (open-pipe* mode "/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" command))

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Kernel modules in initrd
  2018-02-27 20:52                   ` Danny Milosavljevic
@ 2018-02-28 21:49                     ` Ludovic Courtès
  0 siblings, 0 replies; 21+ messages in thread
From: Ludovic Courtès @ 2018-02-28 21:49 UTC (permalink / raw)
  To: Danny Milosavljevic; +Cc: guix-devel

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Tue, 27 Feb 2018 20:32:49 +0100
> Danny Milosavljevic <dannym@scratchpost.org> wrote:
>
>> > > P.S. How come glibc is in the initrd?  Shouldn't guile have statically linked it?
>> > > glibc is like 50000 kiB.  In that case saving 800 kiB is not really worth it...    
>> > 
>> > One of the packages that ends up in the initrd must be dynamically
>> > linked.  You need to find out which one it is.  
>> 
>> It's because of bash-minimal.
>
> And that's because of:
>
> ./gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3/share/guile/2.2/ice-9/popen.scm:  (open-pipe* mode "/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12/bin/bash" "-c" command))

Ouch, indeed!

--8<---------------cut here---------------start------------->8---
$ guix size guile-static-stripped
store item                                                       total    self
/gnu/store/lksm6f1ahvikxnwkfs34sabbb4f7209p-guile-static-stripped-2.2.3    77.0    45.7  59.4%
/gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c    30.3    28.8  37.4%
/gnu/store/05dvazr5wfh7lxx4zi54zfqnx6ha8vxr-bash-static-4.4.12       1.5     1.5   1.9%
/gnu/store/icz3hd36aqpjz5slyp4hhr8wsfbgiml1-bash-minimal-4.4.12     31.3     1.0   1.3%
total: 77.0 MiB
--8<---------------cut here---------------end--------------->8---

How long have we had this?

I’ve fixed it in 63e48300d19b848dfe75880581338c2c73b7b0df, so we’re now
at 45.7 MiB, which is still a lot, but much better.

Ludo'.

^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2018-02-28 21:50 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-22 21:17 Kernel modules in initrd Andreas Enge
2018-02-22 21:29 ` Jan Nieuwenhuizen
2018-02-22 21:44   ` Danny Milosavljevic
2018-02-22 21:34 ` Danny Milosavljevic
2018-02-22 21:50   ` [PATCH] linux-initrd: Add ATA and SAS modules to the default set of modules Danny Milosavljevic
2018-02-27 15:03     ` Ludovic Courtès
2018-02-22 21:53   ` Kernel modules in initrd Andreas Enge
2018-02-22 22:01 ` Mark H Weaver
2018-02-23  1:00   ` Danny Milosavljevic
2018-02-23 14:28     ` Danny Milosavljevic
2018-02-23 23:02       ` Andreas Enge
2018-02-25 11:43         ` Danny Milosavljevic
2018-02-26 15:20           ` Ludovic Courtès
2018-02-26 16:26             ` Danny Milosavljevic
2018-02-27 15:02               ` Ludovic Courtès
2018-02-27 19:32                 ` Danny Milosavljevic
2018-02-27 20:52                   ` Danny Milosavljevic
2018-02-28 21:49                     ` Ludovic Courtès
2018-02-23 22:39   ` Ludovic Courtès
2018-02-24  8:28     ` ng0
2018-02-27 15:04       ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.