Hosting a GuixSD server on commodity hosting platforms, a journey

Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Hello all,

I tried getting GuixSD working on Rackspace Cloud last night.  Ludovic
suggested I write to the list to talk about my success/failure,
regardless of whether I made it or not.  My goal is to get GuixSD hosted
reliably on *some* commodity hosting platform, whether it's Rackspace or
Linode or Digital Ocean or something.  This thread will document my
success/failure.

Well, last night as my first go, I didn't succeed.  I decided to try to
see if I could upload an image generated with "guix system vm-image" and
then upload it and see if I could get it to boot from somewhere in the
UI.  Here are the things I looked at:

 - https://support.rackspace.com/how-to/preparing-an-image-for-import-into-the-rackspace-opencloud/
   Which just says that the file should be in the VHD file format and
   not a lot else (my image didn't exceed limits).
   I converted it like:
     qemu-img convert /tmp/guix-qemu-image.qcow2 -O vpc /tmp/guix-qemu-image.vhd 
 - Trying to figure out how to upload and activate it, I took a look at:
     https://support.rackspace.com/how-to/install-the-swiftly-client-for-cloud-files/
     https://support.rackspace.com/how-to/use-swiftly-to-upload-an-image/
   AFAICT, all these links say are how to upload an image to the
   OpenStack Cloud Files containers.  Okay, well by the time I read this
   I had already uploaded it via the web UI, so I don't see how it would
   be different, with the exception that I didn't use the container name
   "uploaded-images", which it doesn't seem to explicitly require.  So
   effectively, my image was uploaded and resulted in the same place
   afaict, though I just used the web UI rather than the API.
 - Next I tried to boot it.  I went to "Servers -> Cloud Servers" from
   the dropdown menu at the top, then clicked "Create Server".  I
   clicked "Saved"... but there's nothing there.

What am I doing wrong?  I'm not totally sure... I feel like I'm
navigating a jungle out here in the OpenStack / Rackspace docs.  Here's
one thing I found:

  https://community.rackspace.com/products/f/25/t/7186

So I probably need to execute this "import" command.  I guess that's
what's next...

What happens if that doesn't work?  I've been looking at how the Nix
community does things.  Most of the guides seem to involve taking an
existing distribution and converting it.

  https://nixos.org/wiki/Install_NixOS_on_Linode
  https://nixos.org/wiki/Install_NixOS_on_Rackspace_Cloud_Servers

That seems really goofy, but okay, I guess we could do it.

Does anyone else know more about these things than I do?  Your feedback
and experience most welcome!

 - Chris

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Christopher Allan Webber writes:

> What am I doing wrong?  I'm not totally sure... I feel like I'm
> navigating a jungle out here in the OpenStack / Rackspace docs.  Here's
> one thing I found:
>
>   https://community.rackspace.com/products/f/25/t/7186
>
> So I probably need to execute this "import" command.  I guess that's
> what's next...

Well, I did a bit more exploration tonight with *some* progress, but
still not success.  I followed the above link and got a task to generate
an image based off my file.  Looks like the task took!  And it showed up
in "image-list" with the glance command line client.  Sounds like
progress!

And hey, if I try to create the server now via the web UI... if I look
to create an image based off the images list from
"Saved -> Deleted Servers (!!!)" menu, I indeed see my image listed.
Cool!  So I select that and click "create server".

Ok... I wait a bit.  It says it's initializing it!

Uhoh, suddenly the status turns to ERROR.  What's ERROR?  I don't know.
It says ERROR, and it's red.  Hovering over it suggests I ask support.
Hm.

I wonder if I used the Nova command line client if I'd get more
information, or if there's a way to query the API to get more info.

Still, that's *some* progress... I kicked off generating an image
generated via GuixSD, even if it didn't work at all... :)

Relatedly!  User dvc in #guix on freenode suggests looking at
https://www.vultr.com/ which looks quite affordable and hey!  It has a
"custom ISO" option.  If we can convert our USB boot stick thingy
(presumably via xorriso) we could try generating a base server image
from there.  I'd prefer to have a workflow where I go from handing off
something made with "guix system vm-image" to some API, but maybe in the
meanwhile Vultr would be a lower barrier to entry.

In the meanwhile, anyone familiar enough with Nova or Rackspace want to
give me hints on how to find out more about what ERROR means, more
specifically? ;)

Onwards and upwards!
 - Chris

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[ng0]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Onwards and upwards!
>  - Chris

In theory we could also try to init GuixSD from within an Debian
image (server), couldn't we?

So we could also make a list of commonly offered virtual server
distribution choices and try to init GuixSD from within them and
see if it just works.
Is there anyone who tried that before?

-- 
♥Ⓐ  ng0  | ng0.chaosnet.org

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 3633 bytes --]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Christopher Allan Webber writes:
>
>> What am I doing wrong?  I'm not totally sure... I feel like I'm
>> navigating a jungle out here in the OpenStack / Rackspace docs.  Here's
>> one thing I found:
>>
>>   https://community.rackspace.com/products/f/25/t/7186
>>
>> So I probably need to execute this "import" command.  I guess that's
>> what's next...
>
> Well, I did a bit more exploration tonight with *some* progress, but
> still not success.  I followed the above link and got a task to generate
> an image based off my file.  Looks like the task took!  And it showed up
> in "image-list" with the glance command line client.  Sounds like
> progress!
>
> And hey, if I try to create the server now via the web UI... if I look
> to create an image based off the images list from
> "Saved -> Deleted Servers (!!!)" menu, I indeed see my image listed.
> Cool!  So I select that and click "create server".
>
> Ok... I wait a bit.  It says it's initializing it!
>
> Uhoh, suddenly the status turns to ERROR.  What's ERROR?  I don't know.
> It says ERROR, and it's red.  Hovering over it suggests I ask support.
> Hm.
>
> I wonder if I used the Nova command line client if I'd get more
> information, or if there's a way to query the API to get more info.
>
> Still, that's *some* progress... I kicked off generating an image
> generated via GuixSD, even if it didn't work at all... :)
>
> Relatedly!  User dvc in #guix on freenode suggests looking at
> https://www.vultr.com/ which looks quite affordable and hey!  It has a
> "custom ISO" option.  If we can convert our USB boot stick thingy
> (presumably via xorriso) we could try generating a base server image
> from there.  I'd prefer to have a workflow where I go from handing off
> something made with "guix system vm-image" to some API, but maybe in the
> meanwhile Vultr would be a lower barrier to entry.
>
> In the meanwhile, anyone familiar enough with Nova or Rackspace want to
> give me hints on how to find out more about what ERROR means, more
> specifically? ;)

I've had the questionable privilege of working on Openstack for some
time, and indeed have a GuixSD system running on it. I can tell you that
any errors you see are not necessarily representative of what's
happening in the back-end, although it does sound like the Rackspace GUI
just spits out a generic message, instead of a flat-out lie..

The image I used was created by installing a bare-bones GuixSD to an LVM
device (e.g. with Qemu), then dumping this with `dd` before first boot
to create a "RAW" image. RAW is supported (required, actually) by Glance
if Ceph is used as the backing storage, but Rackspace only supports VHD.

I haven't looked into the fstab of the generated VM image, but it may be
hard-coded to '/dev/vda' whereas Xen creates '/dev/xvda' (IIRC). Though
it should at least try to boot if that was the case.

My best advice is to try doing a normal install in a VM from the USB
image (qcow2 probably works) and then convert it to the format your
cloud platform expects, instead of booting the VM image directly. I used
GPT with a "bios_grub" partition FWIW.

Make sure to use (title 'label) and appropriate FS labels, since the
root device path may vary between cloud platforms.

On a vaguely related note, I'm slowly working on a "native" GuixSD
hosting platform, where you will be able to submit (and share!)
configuration files and get a VM and/or disk image back. It's still a
long way off, but I could use some help building the web front-end once
the back-end is ready. Feel free to contact me for more details :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Chris Marusich]

[-- Attachment #1.1: Type: text/plain, Size: 2508 bytes --]

Hi,

Reading Chris' report, I was inspired to try the same thing on EC2.  I
don't know much about Linode or Rackspace, but I do know EC2.  I'm happy
to report that I was successfully in creating a GuixSD EC2 instance.

Here's what I did.  Essentially, all I wanted to do was run "guix system
init" on the root device of an EC2 instance.  I accomplished that by
running "guix system init" on a loopback device (on my laptop), copying
the file which backed the loopback device onto an EC2 instance, writing
the file to an EBS volume attached to the instance, and then replacing
the EC2 instance's root EBS volume with the newly written EBS volume.

Here are the steps I took, in detail.  Hopefully it's clear enough for
someone else to try it out if they want to.

(From the AWS Management Console)

* Launch an instance.  I used an EBS-backed Ubuntu AMI using HVM
  virtualization in the us-west-2 region.  This one: ami-cbd276ab.  It
  needs to be EBS-backed because I want to be able to detach and attach
  the volumes easily; it needs to be HVM virtualization because that way
  it will boot using the GRUB I will install in the GuixSD system's root
  device (paravirtualized EC2 instances follow a different boot
  process).  In the web UI, before launching the instance, I removed the
  ephemeral storage from it (since I didn't need it), and I added an
  extra 30 GiB EBS volume (this will become the GuixSD system's root
  device).  I also gave the instance a name tag to keep track of it in
  the AWS Management Console.  I used an m3.medium type instance, since
  that is the smallest type which would give me consistent performance.
  Here's a quick link to launch the AMI I used:
  https://console.aws.amazon.com/ec2/home?region=us-west-2#launchAmi=ami-cbd276ab

(On my local computer)

* Generate a hashed password for use in the operating system config (see
  "(guile) Encryption" for details).  Here, the password is
  57HdiXRftjV55zpXh5k3 (don't worry, I've changed it :-)):

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (crypt "57HdiXRftjV55zpXh5k3" "$1$JoT5zkpN")
$2 = "$1$JoT5zkpN$1CEM/T/UZ5a4nP1Kh8QoZ/"
scheme@(guile-user)> 
--8<---------------cut here---------------end--------------->8---

* Make the following changes to the "bare-bones" operating system
  configuration file that comes with Guix
  (gnu/system/examples/bare-bones.tmpl), in order to enable remote login
  using that password:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: patch --]
[-- Type: text/x-patch, Size: 1670 bytes --]

--- /home/marusich/guix/gnu/system/examples/bare-bones.tmpl	2016-04-02 13:35:30.226642261 -0700
+++ /home/marusich/bare-bones-with-ssh.scm	2016-11-30 04:59:23.447719406 -0800
@@ -10,11 +10,10 @@
   (timezone "Europe/Berlin")
   (locale "en_US.UTF-8")
 
-  ;; Assuming /dev/sdX is the target hard disk, and "my-root" is
-  ;; the label of the target root file system.
-  (bootloader (grub-configuration (device "/dev/sdX")))
+  ;; Install GRUB to the loopback device so it goes into our disk.img.
+  (bootloader (grub-configuration (device "/dev/loop0")))
   (file-systems (cons (file-system
-                        (device "my-root")
+                        (device "root")
                         (title 'label)
                         (mount-point "/")
                         (type "ext4"))
@@ -25,8 +24,9 @@
   ;; empty password.
   (users (cons (user-account
                 (name "alice")
-                (comment "Bob's sister")
+                (comment "Down the rabbit hole we go")
                 (group "users")
+                (password "$1$JoT5zkpN$1CEM/T/UZ5a4nP1Kh8QoZ/")
 
                 ;; Adding the account to the "wheel" group
                 ;; makes it a sudoer.  Adding it to "audio"
@@ -43,5 +43,7 @@
   ;; Add services to the baseline: a DHCP client and
   ;; an SSH server.
   (services (cons* (dhcp-client-service)
-                   (lsh-service #:port-number 2222)
+                   ;; Don't use lsh since it requires manual
+                   ;; interaction at first boot.
+                   (service openssh-service-type (openssh-configuration))
                    %base-services)))

[-- Attachment #1.3: Type: text/plain, Size: 3599 bytes --]


* Make a sparse file (to save space) to be our disk image:

dd if=/dev/zero of=disk.img bs=1 count=0 seek=30GiB

* Partition it (I suspect that toggling the boot flag is unnecessary,
  but I did it just to be safe):

parted -s disk.img mklabel msdos
parted -s disk.img mkpart primary ext4 10M 100%
parted -s disk.img toggle 1 boot
parted -s disk.img print

* Set up a loopback device:

sudo losetup -P /dev/loop0 disk.img

* Make a file system:

sudo mkfs.ext4 -L root /dev/loop0p1

* Mount it:

sudo mount /dev/loop0p1 /mnt

* Init the system:

sudo guix system init ~/bare-bones-with-ssh.scm /mnt

* Grab some coffee.

* Unmount it:

sudo umount /mnt

* Detach the loopback device:

sudo losetup -d /dev/loop0

* Compress the disk image, so we can copy it to the instance faster.

gzip disk.img

* Copy the image to the instance:

scp -i ~/.ssh/marusich-ec2.pem disk.img.gz ubuntu@ec2-54-214-139-189.us-west-2.compute.amazonaws.com:/tmp

* Log into the instance (Ubuntu EC2 images always use the "ubuntu" user,
  and the marusich-ec2.pem key is the one that I generated when
  launching the instance using the AWS Management Console earlier).

ssh -i ~/.ssh/marusich-ec2.pem ubuntu@ec2-54-214-139-189.us-west-2.compute.amazonaws.com

(While logged into the instance)

* Write the disk image to the target (30 GiB) EBS volume (/dev/xvdf is
  where it's attached):

sudo bash -c 'gzip -d -c /tmp/disk.img.gz > /dev/xvdf'

* Grab another cup of coffee.  This takes a while.

(From the AWS Management Console)

* Stop the instance.

* Detach all its EBS volumes.

* Attach the target (30 GiB) volume to the instance as '/dev/sda1'.
This will actually attach the volume at '/dev/xvda'.  I don't know why
EC2 requires us to attach it as '/dev/sda1', but it does; it's weird.

* Start the instance.

* Now ssh into your new EC2 instance running GuixSD!

ssh alice@ec2-54-189-100-63.us-west-2.compute.amazonaws.com

* From here, you can reconfigure to your heart's content.  I've
  confirmed that running "guix system reconfigure" works.



Some thoughts about the experience:

* The "get console screenshot" and "get system log" features (of Amazon
  EC2) don't work with the GuixSD instance; not sure why, but it made
  troubleshooting a little harder.  Maybe tty settings need tweaking?

* It would have been nice if I could have done this without relying on
  an existing EC2 instance.  In the future, we could consider using the
  EC2 APIs to import an image, similar to what is described here:
  http://docs.aws.amazon.com/vm-import/latest/userguide/import-vm-image.html

* There was some discussion in IRC about whether or not it would make
  sense to have a "base AMI" for GuixSD.  Dave made a good case for
  avoiding that practice [1], which goes against the spirit of GuixSD.
  Instead, for using GuixSD with virtualization services like EC2, in
  the long term, it seems preferable to make it easy to init the system
  you want (including the ssh keys and all) and then import that system
  into the EC2 service (or other virtualization service).  That's
  basically what I did here, albeit via a hacky manual method.

* There is currently no support for declaring a user's authorized_keys
  for SSH.  Just like we have a way to declare a user's password, it
  would be great to have a way to declare a user's authorized_keys.
  There was some discussion about this on IRC also [1], with some
  promising paths forward.

[1] https://gnunet.org/bot/log/guix/2016-11-30

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Marius Bakke writes:

> I've had the questionable privilege of working on Openstack for some
> time, and indeed have a GuixSD system running on it. I can tell you that
> any errors you see are not necessarily representative of what's
> happening in the back-end, although it does sound like the Rackspace GUI
> just spits out a generic message, instead of a flat-out lie..

Yeah it was pretty generic, so I have no idea what happened. :)

> The image I used was created by installing a bare-bones GuixSD to an LVM
> device (e.g. with Qemu), then dumping this with `dd` before first boot
> to create a "RAW" image. RAW is supported (required, actually) by Glance
> if Ceph is used as the backing storage, but Rackspace only supports VHD.

Interesting!

> I haven't looked into the fstab of the generated VM image, but it may be
> hard-coded to '/dev/vda' whereas Xen creates '/dev/xvda' (IIRC). Though
> it should at least try to boot if that was the case.

Yeah, my configuration was set up for /dev/sda I think.  I guess I could
do a system reconfigure switching it out to /dev/vda or whatever at
least minute... I dunno.

> My best advice is to try doing a normal install in a VM from the USB
> image (qcow2 probably works) and then convert it to the format your
> cloud platform expects, instead of booting the VM image directly. I used
> GPT with a "bios_grub" partition FWIW.

Out of curiosity, why do you think this would be an improvement?

> Make sure to use (title 'label) and appropriate FS labels, since the
> root device path may vary between cloud platforms.

Right, though labels might vary between platforms too, correct?

> On a vaguely related note, I'm slowly working on a "native" GuixSD
> hosting platform, where you will be able to submit (and share!)
> configuration files and get a VM and/or disk image back. It's still a
> long way off, but I could use some help building the web front-end once
> the back-end is ready. Feel free to contact me for more details :)

Now this I'm really interested in!  I'll ping you off-list to discuss
this more. :)

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

ng0 writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> Onwards and upwards!
>>  - Chris
>
> In theory we could also try to init GuixSD from within an Debian
> image (server), couldn't we?
>
> So we could also make a list of commonly offered virtual server
> distribution choices and try to init GuixSD from within them and
> see if it just works.
> Is there anyone who tried that before?

That's what most of the NixOS guides do.

(Btw, was moving this from help-guix to guix-devel for a specific
reason?)

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[ng0]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> ng0 writes:
>
>> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>>
>>> Onwards and upwards!
>>>  - Chris
>>
>> In theory we could also try to init GuixSD from within an Debian
>> image (server), couldn't we?
>>
>> So we could also make a list of commonly offered virtual server
>> distribution choices and try to init GuixSD from within them and
>> see if it just works.
>> Is there anyone who tried that before?
>
> That's what most of the NixOS guides do.

Okay, so the goal is to provide something different or better or
less 'hacky' / manual?

> (Btw, was moving this from help-guix to guix-devel for a specific
> reason?)

Sorry, I did not notice the list name, I thought it was on
guix-devel. I don't separate.

-- 
♥Ⓐ  ng0  | ng0.chaosnet.org

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Chris Marusich writes:

> Hi,
>
> Reading Chris' report, I was inspired to try the same thing on EC2.  I
> don't know much about Linode or Rackspace, but I do know EC2.  I'm happy
> to report that I was successfully in creating a GuixSD EC2 instance.
>
> Here's what I did.  Essentially, all I wanted to do was run "guix system
> init" on the root device of an EC2 instance.  I accomplished that by
> running "guix system init" on a loopback device (on my laptop), copying
> the file which backed the loopback device onto an EC2 instance, writing
> the file to an EBS volume attached to the instance, and then replacing
> the EC2 instance's root EBS volume with the newly written EBS volume.

Wow, great!  Thank you for trying it, and documenting it :)

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Christopher Allan Webber writes:

> Relatedly!  User dvc in #guix on freenode suggests looking at
> https://www.vultr.com/ which looks quite affordable and hey!  It has a
> "custom ISO" option.  If we can convert our USB boot stick thingy
> (presumably via xorriso) we could try generating a base server image
> from there.  I'd prefer to have a workflow where I go from handing off
> something made with "guix system vm-image" to some API, but maybe in the
> meanwhile Vultr would be a lower barrier to entry.

I decided to explore this a bit more today, figuring converting Guix's
USB image to .iso couldn't be that hard.  Well, turns out it's another
rabbit hole, and I didn't reach the end of this one yet either!  But I
did make some progress, and thought I'd document what I did.  (Probably
the loopback step can be skipped if we get to making a proper Guix
command out of this using some gexp.  But I decided starting with the
USB image was the path of least resistance.)

Let's start up a little environment:

  $ guix environment --ad-hoc xorriso parted

So the key tool here is xorriso, a rather featureful program for ISO
9960 filesystems (read: data CD-ROMs).  But before we use that, we need
to fetch the usb image (or generate it) and mount it.  Assume you've got
the USB image handy (and uncompressed); then we want to mount it via a
loopback device, but we need to get the right offset to find out what
byte the root partition starts from.  Let's find out where that is.
  
  # We need to find out how many bytes the offset is
  cwebber@oolong:/tmp$ parted guixsd-usb-install-0.11.0.x86_64-linux 
  # This switches it to bytes output
  (parted) unit B
  (parted) print
  Model:  (file)
  Disk /tmp/guixsd-usb-install-0.11.0.x86_64-linux: 943718400B
  Sector size (logical/physical): 512B/512B
  Partition Table: msdos
  Disk Flags: 
  
  Number  Start     End         Size        Type     File system  Flags
   1      1048576B  934281727B  933233152B  primary  ext4         boot
  
  (parted) quit

Ok, so we can take that "start" number (be sure to strip the trailing
'B'):

  $ sudo mount -o ro,loop,offset=1048576 guixsd-usb-install-0.11.0.x86_64-linux /mnt/tmp

Now we can write it to the .iso image, packaging up all those files that
appear in the usb image:

  # flags: verbose, Rock Ridge filesystem info, output path, input directory
  $ sudo xorrisofs -v -R -o ./guixsd-usb-install-0.11.0.x86_64-linux.iso /mnt/tmp/
  GNU xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
  
  Drive current: -outdev 'stdio:./guixsd-usb-install-0.11.0.x86_64-linux.iso'
  Media current: stdio file, overwriteable
  Media status : is blank
  Media summary: 0 sessions, 0 data blocks, 0 data,  242g free
  xorriso : UPDATE : 27100 files added in 1 seconds
  xorriso : UPDATE : 47400 files added in 2 seconds
  xorriso : UPDATE : 50200 files added in 3 seconds
  [.....]
  Written to medium : 351969 sectors at LBA 0
  Writing to 'stdio:./guixsd-usb-install-0.11.0.x86_64-linux.iso' completed successfully.

Except, oops!  This won't boot.  We need to put GRUB on it.

Well here's where I'm stuck for tonight.  I don't know exactly what's
needed; maybe either the -b flag, or --grub2-mbr, or some combination.
The man page is a little bit overwhelming (I mean, xorrisofs is clearly
featureful, credit there):

  https://www.gnu.org/software/xorriso/man_1_xorrisofs.html

But how do I generate the right GRUB stuff to put there?  Can I pull it
off the USB image?  Generate it separately?

This web page is very long but appears to have the appropriate info (and
unfortunately requires running arbitrary javascript to even render):

  http://lukeluo.blogspot.com/2013/06/grub-how-to-2-make-boot-able-iso-with.html

... so I guess the next steps are following roughly what's described at
the bottom of the page?

I wonder how other distros do this step.

The light at the end of this particular rabbit hole feels close, but not
there yet!  As always, external input from those who are more
knowledgable: more than welcome.

 - Chris

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4592 bytes --]

Hi Chris,

Sounds like you've been having fun :-)

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Christopher Allan Webber writes:
>
>> Relatedly!  User dvc in #guix on freenode suggests looking at
>> https://www.vultr.com/ which looks quite affordable and hey!  It has a
>> "custom ISO" option.  If we can convert our USB boot stick thingy
>> (presumably via xorriso) we could try generating a base server image
>> from there.  I'd prefer to have a workflow where I go from handing off
>> something made with "guix system vm-image" to some API, but maybe in the
>> meanwhile Vultr would be a lower barrier to entry.
>
> I decided to explore this a bit more today, figuring converting Guix's
> USB image to .iso couldn't be that hard.  Well, turns out it's another
> rabbit hole, and I didn't reach the end of this one yet either!

You can say that again!  ISOs and bootable CDs are an area that is often
complicated, apparently can be done in multiple ways, and is often
misunderstood or glossed over in most Internet discussions.

> Assume you've got the USB image handy (and uncompressed); then we want
> to mount it via a loopback device, but we need to get the right offset
> to find out what byte the root partition starts from.  Let's find out
> where that is.
> 
>   # We need to find out how many bytes the offset is
>   cwebber@oolong:/tmp$ parted guixsd-usb-install-0.11.0.x86_64-linux 
>   # This switches it to bytes output
>   (parted) unit B
>   (parted) print
>   Model:  (file)
>   Disk /tmp/guixsd-usb-install-0.11.0.x86_64-linux: 943718400B
>   Sector size (logical/physical): 512B/512B
>   Partition Table: msdos
>   Disk Flags: 
>   
>   Number  Start     End         Size        Type     File system  Flags
>    1      1048576B  934281727B  933233152B  primary  ext4         boot
>   
>   (parted) quit
>
> Ok, so we can take that "start" number (be sure to strip the trailing
> 'B'):
>
>   $ sudo mount -o ro,loop,offset=1048576 guixsd-usb-install-0.11.0.x86_64-linux /mnt/tmp

FYI, you can also mount the file on a loopback device, like I did in my
EC2 example.  If you use the -P option to losetup, it'll automatically
create a device file for the partition, which you can then mount.  That
way, there is no need to find/specify the offset.

> Now we can write it to the .iso image, packaging up all those files that
> appear in the usb image:
>
>   # flags: verbose, Rock Ridge filesystem info, output path, input directory
>   $ sudo xorrisofs -v -R -o ./guixsd-usb-install-0.11.0.x86_64-linux.iso /mnt/tmp/
>   GNU xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
>   
>   Drive current: -outdev 'stdio:./guixsd-usb-install-0.11.0.x86_64-linux.iso'
>   Media current: stdio file, overwriteable
>   Media status : is blank
>   Media summary: 0 sessions, 0 data blocks, 0 data,  242g free
>   xorriso : UPDATE : 27100 files added in 1 seconds
>   xorriso : UPDATE : 47400 files added in 2 seconds
>   xorriso : UPDATE : 50200 files added in 3 seconds
>   [.....]
>   Written to medium : 351969 sectors at LBA 0
>   Writing to 'stdio:./guixsd-usb-install-0.11.0.x86_64-linux.iso' completed successfully.
>
> Except, oops!  This won't boot.  We need to put GRUB on it.
>
> Well here's where I'm stuck for tonight.  I don't know exactly what's
> needed; maybe either the -b flag, or --grub2-mbr, or some combination.
> The man page is a little bit overwhelming (I mean, xorrisofs is clearly
> featureful, credit there):
>
>   https://www.gnu.org/software/xorriso/man_1_xorrisofs.html
>
> But how do I generate the right GRUB stuff to put there?  Can I pull it
> off the USB image?  Generate it separately?
>
> This web page is very long but appears to have the appropriate info (and
> unfortunately requires running arbitrary javascript to even render):
>
>   http://lukeluo.blogspot.com/2013/06/grub-how-to-2-make-boot-able-iso-with.html
>
> ... so I guess the next steps are following roughly what's described at
> the bottom of the page?

That is actually the exact webpage I was going to link to you.  What you
need to do (I think) is make the CD bootable, and the common way to do
that, apparently, is to do it via El Torito.  I've never done it,
myself, but my understanding is that that is what needs to be done.

In my opinion, unless the virtualization service provides no other easy
way to import an image for booting, I doubt that making a bootable ISO
is the simplest solution.  But it's good to experiment!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Chris Marusich writes:

> Hi Chris,
>
> Sounds like you've been having fun :-)

Yes, though I'd rather have things working than have fun right now ;)

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> Christopher Allan Webber writes:
>>
>>> Relatedly!  User dvc in #guix on freenode suggests looking at
>>> https://www.vultr.com/ which looks quite affordable and hey!  It has a
>>> "custom ISO" option.  If we can convert our USB boot stick thingy
>>> (presumably via xorriso) we could try generating a base server image
>>> from there.  I'd prefer to have a workflow where I go from handing off
>>> something made with "guix system vm-image" to some API, but maybe in the
>>> meanwhile Vultr would be a lower barrier to entry.
>>
>> I decided to explore this a bit more today, figuring converting Guix's
>> USB image to .iso couldn't be that hard.  Well, turns out it's another
>> rabbit hole, and I didn't reach the end of this one yet either!
>
> You can say that again!  ISOs and bootable CDs are an area that is often
> complicated, apparently can be done in multiple ways, and is often
> misunderstood or glossed over in most Internet discussions.
>
>> Assume you've got the USB image handy (and uncompressed); then we want
>> to mount it via a loopback device, but we need to get the right offset
>> to find out what byte the root partition starts from.  Let's find out
>> where that is.
>> 
>>   # We need to find out how many bytes the offset is
>>   cwebber@oolong:/tmp$ parted guixsd-usb-install-0.11.0.x86_64-linux 
>>   # This switches it to bytes output
>>   (parted) unit B
>>   (parted) print
>>   Model:  (file)
>>   Disk /tmp/guixsd-usb-install-0.11.0.x86_64-linux: 943718400B
>>   Sector size (logical/physical): 512B/512B
>>   Partition Table: msdos
>>   Disk Flags: 
>>   
>>   Number  Start     End         Size        Type     File system  Flags
>>    1      1048576B  934281727B  933233152B  primary  ext4         boot
>>   
>>   (parted) quit
>>
>> Ok, so we can take that "start" number (be sure to strip the trailing
>> 'B'):
>>
>>   $ sudo mount -o ro,loop,offset=1048576 guixsd-usb-install-0.11.0.x86_64-linux /mnt/tmp
>
> FYI, you can also mount the file on a loopback device, like I did in my
> EC2 example.  If you use the -P option to losetup, it'll automatically
> create a device file for the partition, which you can then mount.  That
> way, there is no need to find/specify the offset.

Oh, that's useful... I didn't realize that :)

>> Now we can write it to the .iso image, packaging up all those files that
>> appear in the usb image:
>>
>>   # flags: verbose, Rock Ridge filesystem info, output path, input directory
>>   $ sudo xorrisofs -v -R -o ./guixsd-usb-install-0.11.0.x86_64-linux.iso /mnt/tmp/
>>   GNU xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.
>>   
>>   Drive current: -outdev 'stdio:./guixsd-usb-install-0.11.0.x86_64-linux.iso'
>>   Media current: stdio file, overwriteable
>>   Media status : is blank
>>   Media summary: 0 sessions, 0 data blocks, 0 data,  242g free
>>   xorriso : UPDATE : 27100 files added in 1 seconds
>>   xorriso : UPDATE : 47400 files added in 2 seconds
>>   xorriso : UPDATE : 50200 files added in 3 seconds
>>   [.....]
>>   Written to medium : 351969 sectors at LBA 0
>>   Writing to 'stdio:./guixsd-usb-install-0.11.0.x86_64-linux.iso' completed successfully.
>>
>> Except, oops!  This won't boot.  We need to put GRUB on it.
>>
>> Well here's where I'm stuck for tonight.  I don't know exactly what's
>> needed; maybe either the -b flag, or --grub2-mbr, or some combination.
>> The man page is a little bit overwhelming (I mean, xorrisofs is clearly
>> featureful, credit there):
>>
>>   https://www.gnu.org/software/xorriso/man_1_xorrisofs.html
>>
>> But how do I generate the right GRUB stuff to put there?  Can I pull it
>> off the USB image?  Generate it separately?
>>
>> This web page is very long but appears to have the appropriate info (and
>> unfortunately requires running arbitrary javascript to even render):
>>
>>   http://lukeluo.blogspot.com/2013/06/grub-how-to-2-make-boot-able-iso-with.html
>>
>> ... so I guess the next steps are following roughly what's described at
>> the bottom of the page?
>
> That is actually the exact webpage I was going to link to you.  What you
> need to do (I think) is make the CD bootable, and the common way to do
> that, apparently, is to do it via El Torito.  I've never done it,
> myself, but my understanding is that that is what needs to be done.

Yeah I think we need to generate the Grub configuration, then pass to
Guix somehow.  I don't totally understand how that is all done yet.

> In my opinion, unless the virtualization service provides no other easy
> way to import an image for booting, I doubt that making a bootable ISO
> is the simplest solution.  But it's good to experiment!

Maybe not.  I felt like it might be the easiest path after feeling like
the error message I got from the Rackspace image upload I made was
fairly opaque.  Something where I could watch the output of something
from something resembling a console seemed appealing.

I should probably try to replicate the work you've done on AWS hosting
as a next step.  I'm not wild about AWS, but at this point, maybe having
something working is most important.

I'll re-read what you posted anyway.  Clearly there was some useful info
I missed!

 - CHris

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Baines]

On 02/12/16 04:06, Christopher Allan Webber wrote:
> Except, oops!  This won't boot.  We need to put GRUB on it.
>
> Well here's where I'm stuck for tonight.  I don't know exactly what's
> needed; maybe either the -b flag, or --grub2-mbr, or some combination.
> The man page is a little bit overwhelming (I mean, xorrisofs is clearly
> featureful, credit there):
>
>   https://www.gnu.org/software/xorriso/man_1_xorrisofs.html
>
> But how do I generate the right GRUB stuff to put there?  Can I pull it
> off the USB image?  Generate it separately?
>
> This web page is very long but appears to have the appropriate info (and
> unfortunately requires running arbitrary javascript to even render):
>
>   http://lukeluo.blogspot.com/2013/06/grub-how-to-2-make-boot-able-iso-with.html
>
> ... so I guess the next steps are following roughly what's described at
> the bottom of the page?
>
> I wonder how other distros do this step.

Thanks for looking in to this Chris! I'm using Bytemark for personal 
servers, and have tried and failed to install Guix from a Debian live 
cd. But, they do support installing from arbitrary ISO images, this 
would be great to get working.

The bash script which I guess Nix uses looks quite short [1], you might 
find some useful inspiration there.

1: 
https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.sh

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Hello everyone!  So I got a very nice and useful response about getting
GuixSD working on Rackspace Cloud.  I've gotten permission to post the
conversation verbatim, so I've done so below.  But here's the tl;dr:

 - In order for our image to work, we need to get XenServer and
   nova-agent working and running on it.  DHCP isn't used.
 - Apparently the right format is "VHD in an OVF container"
   but is hard to get right.  I'm assuming we probably won't get
   it right by guessing based on that description.
 - So we probably should use this tool to do our first attempt
   (then figure out from there how to generate such an image
   ourselves without help):
     http://rackerlabs.github.io/boot.rackspace.com/
 - In order to use that, we'll need to supply a .iso, so I guess
   in order to move forward with the Rackspace route, we'll need
   to finish figuring out how to generate a .iso here too!

Thanks again to AlexOughton for the useful feedback.

> [Fri Dec  2 2016]
> <AlexOughton> hi, i just saw your question about GuixSD on #rackspace. you'll
> 	      need to get both the XenServer tools and the nova-agent working,
> 	      as Rackspace Cloud doesn't use DHCP		        [14:52]
> <paroneayea> hi! Ah I see :)
> <paroneayea> I've been documenting my attempt at getting GuixSD on Rackspace
> 	     here:
> 	     http://lists.gnu.org/archive/html/help-guix/2016-11/msg00073.html
> 	     http://lists.gnu.org/archive/html/help-guix/2016-11/msg00074.html
> 								        [14:54]
> <paroneayea> mind if I paste your reply here into the #guix irc channel and/or
> 	     to the mailing list?
> <AlexOughton> sure. i don					        [14:55]
> <AlexOughton> sure. i don't know how far you're going to get with image import
> 	      methods (as this can be flaky, and also requires VERY specific
> 	      format). you might get better luck bootstrapping your own custom
> 	      image this way: http://rackerlabs.github.io/boot.rackspace.com/
> 								        [14:56]
> <AlexOughton> that only provides a few distros, but there are ways of pointing
> 	      iPXE to your own ISO file to then boot that and perform an
> 	      installation					        [14:57]
> <paroneayea> ok, thanks.  Yeah I was having trouble finding any information
> 	     about what the expected format is			        [15:00]
> <paroneayea> and there doesn't seem to be much info
> <AlexOughton> i believe it's VHD in an OVF container.		        [15:02]
> <AlexOughton> i think your success or failure is going to be down to whether
> 	      or not the OS works on XenServer (which is what our hypervisors
> 	      are), and whether or not the XenServer tools work. what kernel
> 	      does guixsd use?
> <paroneayea> linux-libre 4.8.10 is latest I think		        [15:03]
> <paroneayea> you can install other kernels
> <paroneayea> but that's the default
> <AlexOughton> ok, well since it's a Linux my guess would be that it's possible
> 	      to get XenServer tools working. and then the nova-agent is just
> 	      python, so hopefully that's possible too		        [15:04]
> <paroneayea> cool, ok.  I guess we need both nova-agent and XenServer then :)
> <AlexOughton> right. the boot.rackspace link i gave you above has
> 	      documentation for getting both of those working as well
> <paroneayea> cool... well, I was just looking into making a .iso of the boot
> 	     install media yesterday (currently we mostly use a USB install
> 	     stick) so I guess there's extra motivation to do that now :)
> 								        [15:05]
> <paroneayea> thanks for your help, this was a very useful conversation.  Okay
> 	     for me to just paste this conversation verbatim to the mailing
> 	     list?						        [15:06]
> <AlexOughton> yep, no problem. as long as it's understood that i'm just
> 	      answering as an individual, and that this isn't any kind of
> 	      official rackspace support answer. because it's definitely not
> 	      something which rackspace would be able to support for you
> 								        [15:07]
> <paroneayea> understood :)
> <paroneayea> thank you!
> <AlexOughton> you're welcome
> <paroneayea> I really appreciate it :)

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 1842 bytes --]

Chris[es],

[If ‘commodity’ means ‘whatever this OpenStack thing might be’, or
anything involving comparison of pets to cattle, this is not that. But
this caught my eye:]

On 02/12/16 20:51, Christopher Baines wrote:
> Thanks for looking in to this Chris! I'm using Bytemark for personal 
> servers, and have tried and failed to install Guix from a Debian
> live cd.

Interesting. That should work.

I've been running a pure (from scratch) GuixSD VPS[0] for the better
half of a year. I bootstrapped from a provider-supplied SystemRescueCD
image, since it has all required packages pre-installed, but any sane
live system should do.

Partition and format the target device. Mount that. Upload
shiny-new-system.scm. Get your swapon. Bind mount /gnu, /var/guix and
/tmp from temporary directories on /mnt, to prevent ‘guix system init
/mnt’ filling up the RAM disc. Watch the builds scroll by. Reboot. Done.

If I forgot anything, it can't have been much.

A similar approach should work on all hosting providers that support
booting live CDs that can run out of RAM. I suspect one could ‘snapshot’
the base system on providers that support it, to spin up additional
instances faster. I haven't tried; installing from scratch is just as
easy and keeps the bits fresh.

No idea how this relates to ‘commodity hosting’: SystemRescueCD happens
to require console access to set the root password for SSH, but after
that the entire deployment can be scripted. Mine are.

It's as far from underdocumented (to the point of being proprietary)
image layouts and shiny ‘API’s as possible, which is a nice bonus. :-)

Kind regards,

T G-R

[0]: <https://liteserver.nl/>: I happily endorse them, and can't wait to
send them a ‘Please add GuixSD-1.0.iso’ ticket when the time comes.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 476 bytes --]

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Christopher Baines writes:

> On 02/12/16 04:06, Christopher Allan Webber wrote:
>> Except, oops!  This won't boot.  We need to put GRUB on it.
>>
>> Well here's where I'm stuck for tonight.  I don't know exactly what's
>> needed; maybe either the -b flag, or --grub2-mbr, or some combination.
>> The man page is a little bit overwhelming (I mean, xorrisofs is clearly
>> featureful, credit there):
>>
>>   https://www.gnu.org/software/xorriso/man_1_xorrisofs.html
>>
>> But how do I generate the right GRUB stuff to put there?  Can I pull it
>> off the USB image?  Generate it separately?
>>
>> This web page is very long but appears to have the appropriate info (and
>> unfortunately requires running arbitrary javascript to even render):
>>
>>   http://lukeluo.blogspot.com/2013/06/grub-how-to-2-make-boot-able-iso-with.html
>>
>> ... so I guess the next steps are following roughly what's described at
>> the bottom of the page?
>>
>> I wonder how other distros do this step.
>
> Thanks for looking in to this Chris! I'm using Bytemark for personal 
> servers, and have tried and failed to install Guix from a Debian live 
> cd. But, they do support installing from arbitrary ISO images, this 
> would be great to get working.
>
> The bash script which I guess Nix uses looks quite short [1], you might 
> find some useful inspiration there.
>
> 1: 
> https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/make-iso9660-image.sh

That looks VERY helpful, thank you!

Re: Hosting a GuixSD server on commodity hosting platforms, a journey

[Christopher Allan Webber]

Tobias Geerinckx-Rice writes:

> Chris[es],
>
> [If ‘commodity’ means ‘whatever this OpenStack thing might be’, or
> anything involving comparison of pets to cattle, this is not that. But
> this caught my eye:]
>
> On 02/12/16 20:51, Christopher Baines wrote:
>> Thanks for looking in to this Chris! I'm using Bytemark for personal 
>> servers, and have tried and failed to install Guix from a Debian
>> live cd.
>
> Interesting. That should work.
>
> I've been running a pure (from scratch) GuixSD VPS[0] for the better
> half of a year. I bootstrapped from a provider-supplied SystemRescueCD
> image, since it has all required packages pre-installed, but any sane
> live system should do.

Hey great!  And yeah, at this point, maybe if I'm looking at booting
from a .ISO file, while generating a .ISO file is a noble goal still,
I problably should just try installing from another distro.

> Partition and format the target device. Mount that. Upload
> shiny-new-system.scm. Get your swapon. Bind mount /gnu, /var/guix and
> /tmp from temporary directories on /mnt, to prevent ‘guix system init
> /mnt’ filling up the RAM disc. Watch the builds scroll by. Reboot. Done.
>
> If I forgot anything, it can't have been much.
>
> A similar approach should work on all hosting providers that support
> booting live CDs that can run out of RAM. I suspect one could ‘snapshot’
> the base system on providers that support it, to spin up additional
> instances faster. I haven't tried; installing from scratch is just as
> easy and keeps the bits fresh.
>
> No idea how this relates to ‘commodity hosting’: SystemRescueCD happens
> to require console access to set the root password for SSH, but after
> that the entire deployment can be scripted. Mine are.
>
> It's as far from underdocumented (to the point of being proprietary)
> image layouts and shiny ‘API’s as possible, which is a nice bonus. :-)
>
> Kind regards,
>
> T G-R

Sounds good :)

> [0]: <https://liteserver.nl/>: I happily endorse them, and can't wait to
> send them a ‘Please add GuixSD-1.0.iso’ ticket when the time comes.

Wow cool, some of those are quite cheap!