AWS + OpenStack support

AWS + OpenStack support

[Mark Meyer]

Hi list, is anybody interested in having support for running guix on AWS
and/or OpenStack?

I've GuixSD running on AWS. The way to get there was somewhat contrived,
but it worked well.

Basically these `cloud' environments provide some form of instance
configuration. You're supposed to create an image of your OS and it'll
self configure upon (first) boot. This includes more than enabling
DHCP. Generally AWS will provide a so called metadata server. The
important thing serves are your public keys. You'll generally specify a
set of public keys on instance startup.

So I'd like to extend Shepherd with a first-boot service that runs `guix
system reconfigure'. And as a second step I want to include a Guile
package that queries the metadata server. You should be able to do
something like

  (user "guix-sd"
        (public-keys (metadata-get-keys)))

Despite this the only thing that is required to run on AWS is basically
an SSH implementation and a DHCP configuration.

To get GuixSD running on AWS I did the following:

  - Build a disk-image out of my local GuixSD installation, enable
    OpenSSH and DHCP. Create a custom user and set the password.

  - Copy the disk-image to S3

  - Create an IAM role that includes read-only access to S3

  - Start an `Amazon Linux' instance as an intermediate, I used Amazon
    Linux but anything else will just work as well. Attach the IAM role,
    when creating the instance and also attach a 2GB sized EBS volume.

  - Log into the intermediate, download the disk-image from S3 onto
    local disk, then dd the image onto the attached 2GB EBS volume.

  - Do a snapshot of the 2GB volume, then register that as an AMI,
    setting the virtualization type to `HVM'.

You could do without the S3 storage in the middle, but I like it better
this way, since my internet access is not the fastest and you need to
transfer 1~2GB.

Is anybody interested in publicly accessible AMIs for AWS? An AMI is an
Amazon Machine Image. When you got an AMI for your system, you can spin
up an instance with minimal configuration.

I would like to do the aforementioned steps during the next weekend,
which will be a four day weekend in my country, so there's some time to
kill.

Let me know what you think, don't forget to CC me, since I'm not on the
list.

Cheers, Mark

Re: AWS + OpenStack support

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3399 bytes --]

Mark Meyer <mark@ofosos.org> writes:

> Hi list, is anybody interested in having support for running guix on AWS
> and/or OpenStack?

I think it'd be awesome if this were easier to do!  This topic has come
up before:

https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00757.html
https://lists.gnu.org/archive/html/help-guix/2016-11/msg00075.html

Long story short, instead of starting with a base image and modifying it
(e.g., by injecting credentials at first boot via the EC2 metadata
service), one appealing alternative is to use EC2's VM import feature to
actually import precisely the system that you want to launch:

https://aws.amazon.com/ec2/vm-import/

Customizations, such as SSH credentials, would be specified in a GuixSD
operating system configuration file and built into the VM image, so
neither the EC2 metadata service, nor hacks like the "cloud-init" script
used by some distros, would enter into the picture at all.

Some preliminary work in a similar spirit was already done in the branch
'wip-deploy', but I don't think it was EC2-specific in any way.  Perhaps
by looking there, you can find some inspiration?

> Basically these `cloud' environments provide some form of instance
> configuration. You're supposed to create an image of your OS and it'll
> self configure upon (first) boot. This includes more than enabling
> DHCP. Generally AWS will provide a so called metadata server. The
> important thing serves are your public keys. You'll generally specify a
> set of public keys on instance startup.

At first I thought that doing what you suggest was a good idea, but now
I think it'd be better to implement the plan I mentioned above.

> So I'd like to extend Shepherd with a first-boot service that runs `guix
> system reconfigure'. And as a second step I want to include a Guile
> package that queries the metadata server. You should be able to do
> something like
>
>   (user "guix-sd"
>         (public-keys (metadata-get-keys)))

I don't think a "first boot" service like this is a good idea.  To learn
more about why, please read the following conversation between myself
(marusich) and Dave (davexunit) on IRC:

https://gnunet.org/bot/log/guix/2016-11-30

> Is anybody interested in publicly accessible AMIs for AWS? An AMI is an
> Amazon Machine Image. When you got an AMI for your system, you can spin
> up an instance with minimal configuration.

I think it would be better to spend your energy on creating a mechanism
that allows an individual to build a GuixSD image from their own
operating system configuration file, import that into EC2, and then
launch an instance from it.  If such a feature were available in GuixSD,
you could do it once from a desktop/laptop with a slow internet
connection to create a "control server" in the cloud (with a fast
internet connection), and then you could run it from the control server
as needed to quickly spin up whatever other instances you might need.

> I would like to do the aforementioned steps during the next weekend,
> which will be a four day weekend in my country, so there's some time to
> kill.

Even if you just figured out a way to construct a VM image of the kind
that can be imported into EC2, I think it would be a great first step in
the right direction!

Thank you for taking the time to look into this!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: AWS + OpenStack support

[Ludovic Courtès]

Heya,

Mark Meyer <mark@ofosos.org> skribis:

> So I'd like to extend Shepherd with a first-boot service that runs `guix
> system reconfigure'. And as a second step I want to include a Guile
> package that queries the metadata server. You should be able to do
> something like
>
>   (user "guix-sd"
>         (public-keys (metadata-get-keys)))

For this part, you can look at how we register SSH public keys (for lshd
though, not sshd) in this module:

  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/people.scm

I’d like to take this functionality in GuixSD proper and also add it for
sshd.

Thanks,
Ludo’.

Re: AWS + OpenStack support

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Heya,
>
> Mark Meyer <mark@ofosos.org> skribis:
>
>> So I'd like to extend Shepherd with a first-boot service that runs `guix
>> system reconfigure'. And as a second step I want to include a Guile
>> package that queries the metadata server. You should be able to do
>> something like
>>
>>   (user "guix-sd"
>>         (public-keys (metadata-get-keys)))
>
> For this part, you can look at how we register SSH public keys (for lshd
> though, not sshd) in this module:
>
>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/people.scm
>
> I’d like to take this functionality in GuixSD proper and also add it for
> sshd.

I'm not Mark, but I have a question.  When you say "this functionality",
which of the following do you mean?

* A first-boot service that runs 'guix system reconfigure'.

* A Guile module to facilitate querying the EC2 metadata server - this
  would provide the metadata-get-keys procedure as shown above.

* The ability to register a user's SSH public keys by declaring them in
  the operating system configuration file, as shown above.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: AWS + OpenStack support

[Mark Meyer]

>>>>> "Chris" == Chris Marusich <cmmarusich@gmail.com> writes:

    Chris> ludo@gnu.org (Ludovic Courtès) writes:

    Chris> I'm not Mark, but I have a question.  When you say "this
    Chris> functionality", which of the following do you mean?

    Chris> * A first-boot service that runs 'guix system reconfigure'.

    Chris> * A Guile module to facilitate querying the EC2 metadata
    Chris> server - this would provide the metadata-get-keys procedure
    Chris> as shown above.

    Chris> * The ability to register a user's SSH public keys by
    Chris> declaring them in the operating system configuration file, as
    Chris> shown above.

The third one, as far as I can see :)

Cheers, Mark

Re: AWS + OpenStack support

[Mark Meyer]

>>>>> "Ludovic" == Ludovic Courtès <ludo@gnu.org> writes:

    Ludovic> Heya, Mark Meyer <mark@ofosos.org> skribis:

    Ludovic> For this part, you can look at how we register SSH public
    Ludovic> keys (for lshd though, not sshd) in this module:

    Ludovic>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/people.scm

    Ludovic> I’d like to take this functionality in GuixSD proper and
    Ludovic> also add it for sshd.

Can I take this task?

Cheers, Mark

Re: AWS + OpenStack support

[ng0]

Mark Meyer transcribed 0.5K bytes:
> >>>>> "Ludovic" == Ludovic Courtès <ludo@gnu.org> writes:
> 
>     Ludovic> Heya, Mark Meyer <mark@ofosos.org> skribis:
> 
>     Ludovic> For this part, you can look at how we register SSH public
>     Ludovic> keys (for lshd though, not sshd) in this module:
> 
>     Ludovic>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/people.scm
> 
>     Ludovic> I’d like to take this functionality in GuixSD proper and
>     Ludovic> also add it for sshd.
> 
> Can I take this task?
> 
> Cheers, Mark
> 

It would be very welcome. This is currently one of two blocking features
I need until I can introduce GuixSD to IN-Berlin e.V. (among other
services they run virtual servers as a non-profit).

-- 
PGP and more: https://people.pragmatique.xyz/ng0/

Re: AWS + OpenStack support

[Mark Meyer]

>>>>> "ng0" == ng0  <contact.ng0@cryptolab.net> writes:
    ng0> It would be very welcome. This is currently one of two blocking
    ng0> features I need until I can introduce GuixSD to IN-Berlin
    ng0> e.V. (among other services they run virtual servers as a
    ng0> non-profit).

Concerning the OpenSSH configuration, if we don't want to create files
in $HOME, it's possible to pass multiple AuthorizedKeysFile directives
to the server, so this might be:

 AuthorizedKeysFile %h/.ssh/authorized_keys /etc/ssh/%u/authorized_keys

So that guix would control /etc/ssh/%u and you can alternatively add you
own keys to $HOME/.ssh/authorized_keys.

Cheers, Mark

-- 
   Mark Meyer
   mark@ofosos.org

Re: AWS + OpenStack support

[Ludovic Courtès]

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Heya,
>>
>> Mark Meyer <mark@ofosos.org> skribis:
>>
>>> So I'd like to extend Shepherd with a first-boot service that runs `guix
>>> system reconfigure'. And as a second step I want to include a Guile
>>> package that queries the metadata server. You should be able to do
>>> something like
>>>
>>>   (user "guix-sd"
>>>         (public-keys (metadata-get-keys)))
>>
>> For this part, you can look at how we register SSH public keys (for lshd
>> though, not sshd) in this module:
>>
>>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/people.scm
>>
>> I’d like to take this functionality in GuixSD proper and also add it for
>> sshd.
>
> I'm not Mark, but I have a question.  When you say "this functionality",
> which of the following do you mean?
>
> * A first-boot service that runs 'guix system reconfigure'.
>
> * A Guile module to facilitate querying the EC2 metadata server - this
>   would provide the metadata-get-keys procedure as shown above.
>
> * The ability to register a user's SSH public keys by declaring them in
>   the operating system configuration file, as shown above.

I was referring to the ability to register user SSH public keys.  Sorry
for the confusion!

Ludo’.

Re: AWS + OpenStack support

[Ludovic Courtès]

Hello,

Yes, feel free to take the register-user-SSH-keys task!  :-)

Mark Meyer <mark@ofosos.org> skribis:

>>>>>> "ng0" == ng0  <contact.ng0@cryptolab.net> writes:
>     ng0> It would be very welcome. This is currently one of two blocking
>     ng0> features I need until I can introduce GuixSD to IN-Berlin
>     ng0> e.V. (among other services they run virtual servers as a
>     ng0> non-profit).
>
> Concerning the OpenSSH configuration, if we don't want to create files
> in $HOME, it's possible to pass multiple AuthorizedKeysFile directives
> to the server, so this might be:
>
>  AuthorizedKeysFile %h/.ssh/authorized_keys /etc/ssh/%u/authorized_keys
>
> So that guix would control /etc/ssh/%u and you can alternatively add you
> own keys to $HOME/.ssh/authorized_keys.

So the default value for ‘AuthorizedKeysFile’ does not include
/etc/ssh/%u, right?

In that case, we don’t even need to add an entry to /etc and can
directly add something like:

  /gnu/store/…-sshd-authorized-keys

to the config file.

Ludo’.