Any way to boot Guix with root partition elsewhere?

Any way to boot Guix with root partition elsewhere?

[Vladilen Kozin]

Hello guix.

My machine only has USB 2 and rusty old spinners. USB is obviously
slow and guix with its store will quickly destroy any consumer grade
USB stick I insert there. Rusty spinners are so slow I wouldn't want
my worst enemy running their OS on them. So, I plugged PCIe card with
NVMe SSD drive. Problem is, this BIOS won't expose them as available
for booting.

However, one can boot off USB and then "redirect" OS boot to that PCIe
SSD. I've successfully employed just this strategy on FreeBSD, which
lets you specify a different partition for late stage booting (at this
point those NVMe drives have been detected by FreeBSD early stages).

So, I installed Guix on that NVMe and thought I'd be clever and put
grub-bootloader on the USB drive simply by specifying (target
"/dev/sdWHATEVER") in the `bootloader-configuration`. `guix system
init config.scm /mnt` went perfectly fine. Reboot and of course `grub`
can't find the drive by UUID. And `grub rescue` shows that only this
USB is available, sadly it only then occurred to me that `grub` isn't
Linux or Guix so I guess it has no idea how to find those PCIe NVMe
drives.

Is there a way out of this predicament? I can see two options.
(1) can some Linux distro/bootloader do the FreeBSD trick: start on
USB but then chain-boot Guix off that SSD? IIUC grub's chain-boot is
of no help here, cause it simply won't even see that drive;
(2) install and run Guix off USB but somehow ensure that "IO heavy"
bits are mounted from that SSD. There's gotta be some (file-systems
...) combined with the way I `guix system init config.scm /mnt` (or
rather what else I mount under /mnt where) that would achieve this?

In the above I'm going the `manual installation` route i.e. I've been
more or less following
https://guix.gnu.org/manual/en/html_node/Manual-Installation.html
Sadly the GUI based install haven't worked for me even once - I always
find myself staring at some Guile exception or other. Manual worked
every time, though.

Thank you
-- 
Best regards
Vlad Kozin

Re: Any way to boot Guix with root partition elsewhere?

[Joshua Branson]

Vladilen Kozin <vladilen.kozin@gmail.com> writes:

> Hello guix.
>
> Is there a way out of this predicament? I can see two options.
> (1) can some Linux distro/bootloader do the FreeBSD trick: start on
> USB but then chain-boot Guix off that SSD? IIUC grub's chain-boot is
> of no help here, cause it simply won't even see that drive;
> (2) install and run Guix off USB but somehow ensure that "IO heavy"
> bits are mounted from that SSD. There's gotta be some (file-systems
> ...) combined with the way I `guix system init config.scm /mnt` (or
> rather what else I mount under /mnt where) that would achieve this?

Hmm. I'm way out of my depth here...

I think number 1 is your "best" bet.  I've no idea how to do that.

Number 2 is probably your "easiest" option.  /boot/ should be put on the
usb, and I suppose that the / should be on the SSD...You would install
this way.  That's probably how I'd do it.  Or rather that is how I would
try to do it.  :)


This is probably a silly idea...would linuxboot.org help?  You use linux
to boot?


Another silly idea...could you embed linux in your flashrom?  I think
that osboot.org is planning on embedding linux in the flashrom...That
would be pretty awesome.  My flashrom is only 8MB though...


Best of luck!

Joshua

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: Any way to boot Guix with root partition elsewhere?

[Vladilen Kozin]

Well, I went ahead and asked the help-grub@gnu.org but I'm thinking it
maybe difficult or impossible to solve. There may actually be the
"Guix" way to solve this. Here are my current thoughts.

Problem (not specific to Guix):
- we install Guix onto NVMe SSD drive that's plugged via PCIe adaptor,
- BIOS doesn't make those available to boot,
- so you can't boot directly off of that SSD,
- but worse, Grub on any visible drive e.g. USB won't find that SSD either.

IIUC what "solutions" I've seen, they amount to booting off e.g. USB
stick where you have your /boot/grub with grub.cfg BUT with actual
kernel and initrd you want to boot copied over from that SSD drive AND
with whatever kernel parameters your installation suggested. So, I
think roughly this amounts to copying your Guix system's current
kernel and initrd over to USB stick as well as its /boot/grub/grub.cfg
and maybe tweaking it a bit to point to this copied kernel. Or smth
like that.

This obviously defeats the main value proposition of Guix, cause
suddenly no "boot into another system instance", no "roll back".

IMO good solution would be to have Grub load just enough of whatever
kernel is doing to find that drive and make its contents available. I
doubt one can do that, but maybe one can boot into some kernel and
initrd and then "chain" load the Grub on that SSD? I've no clue how to
do that, but I'm sure its possible: kernel is code running in memory,
surely it can "load" and "pass control" over to Grub or smth. If
anyone knows, do tell.

Guix way of solving this could be by coding the above step explicitly
into your config.scm, so that your `guix system reconfigure` having
built everything, copies new kernel and initrd over to USB drive in
addition to earlier kernels etc along with appropriate grub.cfg
(essentially mirroring that bit of your OS on the SSD). Brittle, but
at least you have your earlier systems etc etc.

The whole "boot" business is at the edge of my understanding of how
systems bootstrap and OMG this whole territory is full of arcane
sacred knowledge spread all over the interwebs and most of what you
find is nonsense. Once people who actually know how the sausage is
made retire or die, our civilization is doomed :(

On Sat, 1 May 2021 at 04:07, Joshua Branson <jbranso@dismail.de> wrote:
>
> Vladilen Kozin <vladilen.kozin@gmail.com> writes:
>
> > Hello guix.
> >
> > Is there a way out of this predicament? I can see two options.
> > (1) can some Linux distro/bootloader do the FreeBSD trick: start on
> > USB but then chain-boot Guix off that SSD? IIUC grub's chain-boot is
> > of no help here, cause it simply won't even see that drive;
> > (2) install and run Guix off USB but somehow ensure that "IO heavy"
> > bits are mounted from that SSD. There's gotta be some (file-systems
> > ...) combined with the way I `guix system init config.scm /mnt` (or
> > rather what else I mount under /mnt where) that would achieve this?
>
> Hmm. I'm way out of my depth here...
>
> I think number 1 is your "best" bet.  I've no idea how to do that.
>
> Number 2 is probably your "easiest" option.  /boot/ should be put on the
> usb, and I suppose that the / should be on the SSD...You would install
> this way.  That's probably how I'd do it.  Or rather that is how I would
> try to do it.  :)
>
>
> This is probably a silly idea...would linuxboot.org help?  You use linux
> to boot?
>
>
> Another silly idea...could you embed linux in your flashrom?  I think
> that osboot.org is planning on embedding linux in the flashrom...That
> would be pretty awesome.  My flashrom is only 8MB though...
>
>
> Best of luck!
>
> Joshua
>
> --
> Joshua Branson (joshuaBPMan in #guix)
> Sent from Emacs and Gnus
>   https://gnucode.me
>   https://video.hardlimit.com/accounts/joshua_branson/video-channels
>   https://propernaming.org
>   "You can have whatever you want, as long as you help
> enough other people get what they want." - Zig Ziglar



-- 
Best regards
Vlad Kozin

Re: Any way to boot Guix with root partition elsewhere?

[Joshua Branson]

Vladilen Kozin <vladilen.kozin@gmail.com> writes:

> IMO good solution would be to have Grub load just enough of whatever
> kernel is doing to find that drive and make its contents available. I
> doubt one can do that, but maybe one can boot into some kernel and
> initrd and then "chain" load the Grub on that SSD? I've no clue how to
> do that, but I'm sure its possible: kernel is code running in memory,
> surely it can "load" and "pass control" over to Grub or smth. If
> anyone knows, do tell.

I think you just don't want to use the "slow" hard drive right?  I
personally use a hard drive, and I am content with the performance.  I
just prefer to use the old lenovo T400.  I think it cost me $300 USD to
assemble it.  Not to shabby.  :)

-- 
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar
  

Re: Any way to boot Guix with root partition elsewhere?

[Jonathan McHugh]

A shot in the dark - but I know that there have been some activity with
regards to using Lua in the kernel:
=> https://github.com/luainkernel
=> lwn.net/Articles/830154/

If thats not helpful hopefully it can give you some ideas.

Has Guile done much activity interacting with the kernel?


Jonathan


Vladilen Kozin <vladilen.kozin@gmail.com> writes:

> Well, I went ahead and asked the help-grub@gnu.org but I'm thinking it
> maybe difficult or impossible to solve. There may actually be the
> "Guix" way to solve this. Here are my current thoughts.
>
> Problem (not specific to Guix):
> - we install Guix onto NVMe SSD drive that's plugged via PCIe adaptor,
> - BIOS doesn't make those available to boot,
> - so you can't boot directly off of that SSD,
> - but worse, Grub on any visible drive e.g. USB won't find that SSD either.
>
> IIUC what "solutions" I've seen, they amount to booting off e.g. USB
> stick where you have your /boot/grub with grub.cfg BUT with actual
> kernel and initrd you want to boot copied over from that SSD drive AND
> with whatever kernel parameters your installation suggested. So, I
> think roughly this amounts to copying your Guix system's current
> kernel and initrd over to USB stick as well as its /boot/grub/grub.cfg
> and maybe tweaking it a bit to point to this copied kernel. Or smth
> like that.
>
> This obviously defeats the main value proposition of Guix, cause
> suddenly no "boot into another system instance", no "roll back".
>
> IMO good solution would be to have Grub load just enough of whatever
> kernel is doing to find that drive and make its contents available. I
> doubt one can do that, but maybe one can boot into some kernel and
> initrd and then "chain" load the Grub on that SSD? I've no clue how to
> do that, but I'm sure its possible: kernel is code running in memory,
> surely it can "load" and "pass control" over to Grub or smth. If
> anyone knows, do tell.
>
> Guix way of solving this could be by coding the above step explicitly
> into your config.scm, so that your `guix system reconfigure` having
> built everything, copies new kernel and initrd over to USB drive in
> addition to earlier kernels etc along with appropriate grub.cfg
> (essentially mirroring that bit of your OS on the SSD). Brittle, but
> at least you have your earlier systems etc etc.
>
> The whole "boot" business is at the edge of my understanding of how
> systems bootstrap and OMG this whole territory is full of arcane
> sacred knowledge spread all over the interwebs and most of what you
> find is nonsense. Once people who actually know how the sausage is
> made retire or die, our civilization is doomed :(
>
> On Sat, 1 May 2021 at 04:07, Joshua Branson <jbranso@dismail.de> wrote:
>>
>> Vladilen Kozin <vladilen.kozin@gmail.com> writes:
>>
>> > Hello guix.
>> >
>> > Is there a way out of this predicament? I can see two options.
>> > (1) can some Linux distro/bootloader do the FreeBSD trick: start on
>> > USB but then chain-boot Guix off that SSD? IIUC grub's chain-boot is
>> > of no help here, cause it simply won't even see that drive;
>> > (2) install and run Guix off USB but somehow ensure that "IO heavy"
>> > bits are mounted from that SSD. There's gotta be some (file-systems
>> > ...) combined with the way I `guix system init config.scm /mnt` (or
>> > rather what else I mount under /mnt where) that would achieve this?
>>
>> Hmm. I'm way out of my depth here...
>>
>> I think number 1 is your "best" bet.  I've no idea how to do that.
>>
>> Number 2 is probably your "easiest" option.  /boot/ should be put on the
>> usb, and I suppose that the / should be on the SSD...You would install
>> this way.  That's probably how I'd do it.  Or rather that is how I would
>> try to do it.  :)
>>
>>
>> This is probably a silly idea...would linuxboot.org help?  You use linux
>> to boot?
>>
>>
>> Another silly idea...could you embed linux in your flashrom?  I think
>> that osboot.org is planning on embedding linux in the flashrom...That
>> would be pretty awesome.  My flashrom is only 8MB though...
>>
>>
>> Best of luck!
>>
>> Joshua
>>
>> --
>> Joshua Branson (joshuaBPMan in #guix)
>> Sent from Emacs and Gnus
>>   https://gnucode.me
>>   https://video.hardlimit.com/accounts/joshua_branson/video-channels
>>   https://propernaming.org
>>   "You can have whatever you want, as long as you help
>> enough other people get what they want." - Zig Ziglar


-- 
Jonathan McHugh
indieterminacy@libre.brussels

support split /boot partition

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2058 bytes --]

On 2021-04-30, Joshua Branson wrote:
> Vladilen Kozin <vladilen.kozin@gmail.com> writes:
>> Is there a way out of this predicament? I can see two options.
>> (1) can some Linux distro/bootloader do the FreeBSD trick: start on
>> USB but then chain-boot Guix off that SSD? IIUC grub's chain-boot is
>> of no help here, cause it simply won't even see that drive;
>> (2) install and run Guix off USB but somehow ensure that "IO heavy"
>> bits are mounted from that SSD. There's gotta be some (file-systems
>> ...) combined with the way I `guix system init config.scm /mnt` (or
>> rather what else I mount under /mnt where) that would achieve this?
...
> I think number 1 is your "best" bet.  I've no idea how to do that.
>
> Number 2 is probably your "easiest" option.  /boot/ should be put on the
> usb, and I suppose that the / should be on the SSD...You would install
> this way.  That's probably how I'd do it.  Or rather that is how I would
> try to do it.  :)

Unfortunately, guix doesn't currently support booting off of a separate
/boot partition, since the kernel and initrd are in /gnu/store; your
bootloader needs to be able to mount the partition that /gnu/store is
located on.

The workaround would be to manually copy all files mentioned in grub.cfg
(kernel, initrd, possibly others) into a partition somewhere on boot
media, and tweak the grub.cfg appropriately...


There are several cases where this sort of thing would be desireable:

* The above scenario; the system does not expose an NVMe drive from EFI
  or BIOS.

* Using u-boot and you want root on lvm, raid, encryption, etc. which
  u-boot does not support


I've been meaning to file a bug on this for ages; I have BCCed bug-guix
now which will hopefully work. :)


Another more complicated option would be to use kexec; where you boot
one kernel and maybe a minimal rootfs, and load a new kernel and initrd
into ram, which uses the kernel drivers and then runs "kexec" to switch
to the newly loaded kernel... not sure how reliable that is, but it is
possible.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: support split /boot partition

[Vincent Legoll]

Hello,

On Sun, May 2, 2021 at 11:17 PM Vagrant Cascadian <vagrant@debian.org> wrote:
> Another more complicated option would be to use kexec; where you boot
> one kernel and maybe a minimal rootfs, and load a new kernel and initrd
> into ram, which uses the kernel drivers and then runs "kexec" to switch
> to the newly loaded kernel... not sure how reliable that is, but it is
> possible.

This is actually usable and used, see petitboot or linuxboot.

https://github.com/open-power/petitboot
https://www.linuxboot.org

-- 
Vincent Legoll