extending initrd

extending initrd

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 303 bytes --]

Hi,

I'm playing a bit with initrd and I miss there a way, how to add
additional content to the image (busybox in my case now). Is there
really no way yet how to do that?

In expression->initrd it refers to closure yet I fail to find how is
it found/constructed...

Thanks in advance for pointers.

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Tomáš Čech]

* gnu/system/linux-initrd.scm(base-initrd): Add `extra-packages' parameter.
---
 gnu/system/linux-initrd.scm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 4934c92..1ba6692 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -139,6 +139,7 @@ MODULES and taken from LINUX."
                       (virtio? #t)
                       volatile-root?
                       (linux-modules #f)
+		      (extra-packages '())
                       (extra-modules '()))
   "Return a monadic derivation that builds a generic initrd, with kernel
 modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
@@ -209,7 +210,8 @@ loaded at boot time in the order in which they appear."
             '())
       ,@(if volatile-root?
             (list unionfs-fuse/static)
-            '())))
+            '())
+      ,@extra-packages))
 
   (define device-mapping-commands
     ;; List of gexps to open the mapped devices.
-- 
2.9.2

Re: [PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

Ping after week - is there interest in this patch?

I know that

- it doesn't apply cleanly as it depends on the other one I sent later
  and was discussed
  
- documentation needs to be adjusted as well

- it's a bit corner case but improves flexibility of the configuration
  and is useful for debugging, early HW init, fixing filesystems from
  initial tmpfs etc.

WDYT?

S_W

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Andy Wingo]

On Mon 01 Aug 2016 12:39, Tomáš Čech <sleep_walker@gnu.org> writes:

> Ping after week - is there interest in this patch?
>
> I know that
>
> - it doesn't apply cleanly as it depends on the other one I sent later
>  and was discussed
>  - documentation needs to be adjusted as well
>
> - it's a bit corner case but improves flexibility of the configuration
>  and is useful for debugging, early HW init, fixing filesystems from
>  initial tmpfs etc.
>
> WDYT?

Makes sense to me; needs a documentation addition though.

Andy

extra-packages - second wave

[Tomáš Čech]

I have made the patch applicable and added documentation in a way I find
useful. As English is not my mother tounge, it may need to be corrected.

Thanks in advance for review.

S_W

[PATCH 1/2] linux-initrd: Introduce way to add more packages to initrd.

[Tomáš Čech]

* gnu/system/linux-initrd.scm(base-initrd): Add `extra-packages' parameter.
---
 gnu/system/linux-initrd.scm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index bbaa5c0..1676684 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -138,6 +138,7 @@ MODULES and taken from LINUX."
                       qemu-networking?
                       (virtio? #t)
                       volatile-root?
+                      (extra-packages '())
                       (extra-modules '()))
   "Return a monadic derivation that builds a generic initrd, with kernel
 modules taken from LINUX.  FILE-SYSTEMS is a list of file-systems to be
@@ -208,7 +209,8 @@ loaded at boot time in the order in which they appear."
             '())
       ,@(if volatile-root?
             (list unionfs-fuse/static)
-            '())))
+            '())
+      ,@extra-packages))
 
   (define device-mapping-commands
     ;; List of gexps to open the mapped devices.
-- 
2.9.2

[PATCH 2/2] doc: Document extra-packages argument of base-initrd.

[Tomáš Čech]

* doc/guix.texi(Initial RAM Disk): Describe extra-modules parameter in
  definition of base-initrd. Mention it under example of use of
  extra-packges.
---
 doc/guix.texi | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 786fe55..27a4cea 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -10183,6 +10183,9 @@ system declaration like this:
                  rest)))
 @end example
 
+In a similar way you can add programs to packed into initrd using
+@code{#:extra-packages}.
+
 The @code{base-initrd} procedure also handles common use cases that
 involves using the system as a QEMU guest, or as a ``live'' system with
 volatile root file system.
@@ -10233,7 +10236,7 @@ further.
 
 @deffn {Monadic Procedure} base-initrd @var{file-systems} @
        [#:qemu-networking? #f] [#:virtio? #t] [#:volatile-root? #f] @
-       [#:extra-modules '()] [#:mapped-devices '()]
+       [#:extra-packages '()] [#:extra-modules '()] [#:mapped-devices '()]
 Return a monadic derivation that builds a generic initrd.  @var{file-systems} is
 a list of file systems to be mounted by the initrd, possibly in addition to
 the root file system specified on the kernel command line via @code{--root}.
@@ -10251,6 +10254,11 @@ The initrd is automatically populated with all the kernel modules necessary
 for @var{file-systems} and for the given options.  However, additional kernel
 modules can be listed in @var{extra-modules}.  They will be added to the initrd, and
 loaded at boot time in the order in which they appear.
+
+Additional packages can be added through @var{extra-packages}. This is
+useful mostly for development and debugging purposes but it can also be
+used to extend set of commands available in case of interactive
+recovery.
 @end deffn
 
 Needless to say, the initrds we produce and use embed a
-- 
2.9.2

Re: [PATCH 2/2] doc: Document extra-packages argument of base-initrd.

[Vincent Legoll]

On Tue, Aug 2, 2016 at 9:43 AM, Tomáš Čech <sleep_walker@gnu.org> wrote:
> * doc/guix.texi(Initial RAM Disk): Describe extra-modules parameter in
>   definition of base-initrd. Mention it under example of use of
>   extra-packges.
> ---
>  doc/guix.texi | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/doc/guix.texi b/doc/guix.texi
> index 786fe55..27a4cea 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -10183,6 +10183,9 @@ system declaration like this:
>                   rest)))
>  @end example
>
> +In a similar way you can add programs to packed into initrd using
> +@code{#:extra-packages}.

"to pack in the" maybe ?

>  The @code{base-initrd} procedure also handles common use cases that
>  involves using the system as a QEMU guest, or as a ``live'' system with
>  volatile root file system.
> @@ -10233,7 +10236,7 @@ further.
>
>  @deffn {Monadic Procedure} base-initrd @var{file-systems} @
>         [#:qemu-networking? #f] [#:virtio? #t] [#:volatile-root? #f] @
> -       [#:extra-modules '()] [#:mapped-devices '()]
> +       [#:extra-packages '()] [#:extra-modules '()] [#:mapped-devices '()]
>  Return a monadic derivation that builds a generic initrd.  @var{file-systems} is
>  a list of file systems to be mounted by the initrd, possibly in addition to
>  the root file system specified on the kernel command line via @code{--root}.
> @@ -10251,6 +10254,11 @@ The initrd is automatically populated with all the kernel modules necessary
>  for @var{file-systems} and for the given options.  However, additional kernel
>  modules can be listed in @var{extra-modules}.  They will be added to the initrd, and
>  loaded at boot time in the order in which they appear.
> +
> +Additional packages can be added through @var{extra-packages}. This is
> +useful mostly for development and debugging purposes but it can also be
> +used to extend set of commands available in case of interactive
> +recovery.
>  @end deffn

"to extend the set"

-- 
Vincent Legoll

Re: [PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Ludovic Courtès]

Hello,

Tomáš Čech <sleep_walker@gnu.org> skribis:

> Ping after week - is there interest in this patch?

I didn’t comment on the patch specifically because I thought my reply
might make you change your mind.  ;-)

  https://lists.gnu.org/archive/html/guix-devel/2016-07/msg01241.html

All the packages used by the initrd are automatically part of the
initrd.  The proposed patch would allow people to add unused packages to
the initrd.

Could you explain how/when this would be used?  Maybe as commands for
use by Bournish when it’s used as a rescue shell?

Thanks,
Ludo’.

Re: [PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 1631 bytes --]

On Mon, Aug 01, 2016 at 06:09:12PM +0200, Ludovic Courtès wrote:
>Hello,
>
>Tomáš Čech <sleep_walker@gnu.org> skribis:
>
>> Ping after week - is there interest in this patch?
>
>I didn’t comment on the patch specifically because I thought my reply
>might make you change your mind.  ;-)
>
>  https://lists.gnu.org/archive/html/guix-devel/2016-07/msg01241.html

Thanks for that, I'm still trying to digest that bite. I'm sorry for
being a bit slow on Guile.

>All the packages used by the initrd are automatically part of the
>initrd.  The proposed patch would allow people to add unused packages to
>the initrd.

It is for the packages which you may want to use interactivelly in
case of failure or for some extra initrd hacking you may not want/be
able to write in Guile.

Features like
- extra authentication
- full disk encryption
- root on NFS
- LVM :)
- ...

when kernel+initrd is booted from another source.

>Could you explain how/when this would be used?  Maybe as commands for
>use by Bournish when it’s used as a rescue shell?

I agree that it is more for debugging and to balance my inability to
express it in Guile but it lowers the barrier a bit.

Bournish is too young to rely on it. I miss pipes, accessing files in
different directories or `ls' with wildcards. I can put in minimal
static busybox which is more than sufficient for rescue, problem
analysis or even data recovery.

I like the idea of Bournish but I'd rather have an alternative
until it is more capable.

And yes, it could be also used from Bournish in interactive session as
a command.

S_W

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Ludovic Courtès]

Hello,

Tomáš Čech <sleep_walker@gnu.org> skribis:

> On Mon, Aug 01, 2016 at 06:09:12PM +0200, Ludovic Courtès wrote:
>>Hello,
>>
>>Tomáš Čech <sleep_walker@gnu.org> skribis:
>>
>>> Ping after week - is there interest in this patch?
>>
>>I didn’t comment on the patch specifically because I thought my reply
>>might make you change your mind.  ;-)
>>
>>  https://lists.gnu.org/archive/html/guix-devel/2016-07/msg01241.html
>
> Thanks for that, I'm still trying to digest that bite. I'm sorry for
> being a bit slow on Guile.

No problem.  Since you pinged, I thought you had digested it.  ;-)

>>All the packages used by the initrd are automatically part of the
>>initrd.  The proposed patch would allow people to add unused packages to
>>the initrd.
>
> It is for the packages which you may want to use interactivelly in
> case of failure or for some extra initrd hacking you may not want/be
> able to write in Guile.
>
> Features like
> - extra authentication
> - full disk encryption
> - root on NFS
> - LVM :)
> - ...

OK but if you need these packages, for instance because you have a LUKS
boot device, they’ll already be in the initrd.  No need to manually list
them in #:extra-packages.

>>Could you explain how/when this would be used?  Maybe as commands for
>>use by Bournish when it’s used as a rescue shell?
>
> I agree that it is more for debugging and to balance my inability to
> express it in Guile but it lowers the barrier a bit.
>
> Bournish is too young to rely on it. I miss pipes, accessing files in
> different directories or `ls' with wildcards. I can put in minimal
> static busybox which is more than sufficient for rescue, problem
> analysis or even data recovery.
>
> I like the idea of Bournish but I'd rather have an alternative
> until it is more capable.

I agree, but hopefully, you don’t run into Bournish too often?

I guess my main concern (again, as a lazy maintainer) is the cost of
turning ‘base-initrd’ into a kitchen sink, as discussed in the other
thread about #:extra-modules.

I would prefer to provide simple tools that people can build upon, like
‘expression->initrd’ or the ‘raw-initrd’ procedure I proposed, than
trying to come up with a one-size-fits-all procedure with many
parameters.

WDYT?

Thanks,
Ludo’.

Re: [PATCH] linux-initrd: Introduce way to add more packages to initrd.

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 3025 bytes --]

On Tue, Aug 02, 2016 at 02:53:58PM +0200, Ludovic Courtès wrote:
>Hello,
>
>Tomáš Čech <sleep_walker@gnu.org> skribis:
>>>All the packages used by the initrd are automatically part of the
>>>initrd.  The proposed patch would allow people to add unused packages to
>>>the initrd.
>>
>> It is for the packages which you may want to use interactivelly in
>> case of failure or for some extra initrd hacking you may not want/be
>> able to write in Guile.
>>
>> Features like
>> - extra authentication
>> - full disk encryption
>> - root on NFS
>> - LVM :)
>> - ...
>
>OK but if you need these packages, for instance because you have a LUKS
>boot device, they’ll already be in the initrd.  No need to manually list
>them in #:extra-packages.

That is correct only for cases which are already handle. Last time I
booted from LVM it didn't work. IOW your answer is valid for user but
not for (potential) contributor.

This can help me (and possibly others) to write it.

Otherwise it is hard to write for Guix in Guile because it is hard to
write for Guix in Guile. My solution is not perfect but helps.

>>>Could you explain how/when this would be used?  Maybe as commands for
>>>use by Bournish when it’s used as a rescue shell?
>>
>> I agree that it is more for debugging and to balance my inability to
>> express it in Guile but it lowers the barrier a bit.
>>
>> Bournish is too young to rely on it. I miss pipes, accessing files in
>> different directories or `ls' with wildcards. I can put in minimal
>> static busybox which is more than sufficient for rescue, problem
>> analysis or even data recovery.
>>
>> I like the idea of Bournish but I'd rather have an alternative
>> until it is more capable.
>
>I agree, but hopefully, you don’t run into Bournish too often?

Of course I did until I used exactly this patch and was able to solve
my problem.

>I guess my main concern (again, as a lazy maintainer) is the cost of
>turning ‘base-initrd’ into a kitchen sink, as discussed in the other
>thread about #:extra-modules.
>
>I would prefer to provide simple tools that people can build upon, like
>‘expression->initrd’ or the ‘raw-initrd’ procedure I proposed, than
>trying to come up with a one-size-fits-all procedure with many
>parameters.

I know that your approach is cleaner and nicer for any Guile
programmer. On the other hand it can be harder to use for the rest of
the world.

Kitchen sink issue is matter of configuration and we're all adults
here, we can make our decisions.

I'm not ignoring your proposal but

- I haven't seen a way how it could help me in previous thread
- I'm afraid I can't deliver you such solution until I improve my
  Guile skills.

I'm fine with keeping my patches out of tree (they can be found in
devel mailing list if anyone is interested in it), I just don't think
that I'm the only one facing this sort of problems.

I'm sorry I didn't provide you more constructive answer.

S_W

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: extending initrd

[Ludovic Courtès]

Hi,

Tomáš Čech <sleep_walker@gnu.org> skribis:

> I'm playing a bit with initrd and I miss there a way, how to add
> additional content to the image (busybox in my case now). Is there
> really no way yet how to do that?

The initrd automatically contains everything the given gexp refers to:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> ,use(gnu system linux-initrd)
scheme@(guile-user)> ,use(gnu packages busybox)
scheme@(guile-user)> ,enter-store-monad
store-monad@(guile-user) [1]> (expression->initrd #~(execl (string-append #$busybox "/bin/uname") "uname" "-a"))
$2 = #<derivation /gnu/store/ns4abqm8llcm06r9wg8rrnaj5axvg3la-guile-initrd.drv => /gnu/store/5m2vp3z30b8f9yxv6yqa0x5imssl2qvr-guile-initrd 377cc30>
store-monad@(guile-user) [1]> (built-derivations (list $2))
[...]
--8<---------------cut here---------------end--------------->8---

… and then:

--8<---------------cut here---------------start------------->8---
$ gunzip < /gnu/store/5m2vp3z30b8f9yxv6yqa0x5imssl2qvr-guile-initrd/initrd | cpio -tv|grep busybox.*bin/uname
lrwxrwxrwx   1 root     root            7 Jan  1  1970 ./gnu/store/0rgimvxi572zncpr87q3867hkwkfmlva-busybox-1.25.0/bin/uname -> busybox
106143 blocks
--8<---------------cut here---------------end--------------->8---

> In expression->initrd it refers to closure yet I fail to find how is
> it found/constructed...

If you look at the definition of ‘expression->initrd’, you’ll see that
it turns the given gexp into a script using ‘gexp->script’.  In the
example above, said script refers to Busybox, because our expression
refers to Busybox.

Then, ‘expression->initrd’ queries the closure of this script via
#:references-graphs and populates the initrd with exclusively what’s in
that closure.  In this example, the closure includes Busybox and Guile.

Very short introduction, but I hope it helps!

Ludo’.