[Mark Meyer] Re: AWS + OpenStack support

[Mark Meyer] Re: AWS + OpenStack support

[Mark Meyer]

[-- Attachment #1: Type: text/plain, Size: 31 bytes --]

Sorry, didn't go to the list.


[-- Attachment #2: Type: message/rfc822, Size: 3370 bytes --]

From: Mark Meyer <mark@ofosos.org>
To: Chris Marusich <cmmarusich@gmail.com>
Subject: Re: AWS + OpenStack support
Date: Tue, 11 Apr 2017 07:45:08 +0200
Message-ID: <878tn7h50b.fsf@ofosos.org>

>>>>> "Chris" == Chris Marusich <cmmarusich@gmail.com> writes:

    Chris> Mark Meyer <mark@ofosos.org> writes:

    Chris> I think it'd be awesome if this were easier to do!  This
    Chris> topic has come up before:

    Chris> https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00757.html
    Chris> https://lists.gnu.org/archive/html/help-guix/2016-11/msg00075.html

Ah. I completely missed the latter discussion.

    Chris> Long story short, instead of starting with a base image and
    Chris> modifying it (e.g., by injecting credentials at first boot
    Chris> via the EC2 metadata service), one appealing alternative is
    Chris> to use EC2's VM import feature to actually import precisely
    Chris> the system that you want to launch:

    Chris> https://aws.amazon.com/ec2/vm-import/

Which does not work with GuixSD (tried it). Apparently it looks into the
image an expects stuff like fstab. I find it not very trust building
that it actually inspects the image.

    Chris> Customizations, such as SSH credentials, would be specified
    Chris> in a GuixSD operating system configuration file and built
    Chris> into the VM image, so neither the EC2 metadata service, nor
    Chris> hacks like the "cloud-init" script used by some distros,
    Chris> would enter into the picture at all.

    Chris> Some preliminary work in a similar spirit was already done in
    Chris> the branch 'wip-deploy', but I don't think it was
    Chris> EC2-specific in any way.  Perhaps by looking there, you can
    Chris> find some inspiration?

Here the immediate downside would be that stuff like auto-scaling does
not work out of the box. Which some people consider one of the selling
features of AWS, the prices for VM hosting being rather high.

    Chris> I think it would be better to spend your energy on creating a
    Chris> mechanism that allows an individual to build a GuixSD image
    Chris> from their own operating system configuration file, import
    Chris> that into EC2, and then launch an instance from it.  If such
    Chris> a feature were available in GuixSD, you could do it once from
    Chris> a desktop/laptop with a slow internet connection to create a
    Chris> "control server" in the cloud (with a fast internet
    Chris> connection), and then you could run it from the control
    Chris> server as needed to quickly spin up whatever other instances
    Chris> you might need.

I think the above steps could be shortened somewhat and automated, if
you know you're running on ec2.

I don't see a way to cleanly import an image into AWS. This is however
different for OpenStack, there you have an image service that does just
what we need.

I'll try my hand at optimizing these steps on the weekend.

Cheer, Mark
-- 
   Mark Meyer
   mark@ofosos.org

[-- Attachment #3: Type: text/plain, Size: 39 bytes --]



-- 
   Mark Meyer
   mark@ofosos.org

Re: [Mark Meyer] Re: AWS + OpenStack support

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4259 bytes --]

Mark Meyer <mark@ofosos.org> writes:

>     Chris> Long story short, instead of starting with a base image and
>     Chris> modifying it (e.g., by injecting credentials at first boot
>     Chris> via the EC2 metadata service), one appealing alternative is
>     Chris> to use EC2's VM import feature to actually import precisely
>     Chris> the system that you want to launch:
>
>     Chris> https://aws.amazon.com/ec2/vm-import/
>
> Which does not work with GuixSD (tried it). Apparently it looks into the
> image an expects stuff like fstab. I find it not very trust building
> that it actually inspects the image.

Yes, we would need to write some code that creates the kind of image (
that EC2 expects.  The documentation claims that "raw" format is
supported; I guess they have some unspoken requirements (like the fstab
thing you mentioned) that complicate matters?

https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#prerequisites-image

For what it's worth, if you have a support contract (even a low tier
one) with AWS, their support engineers can help answer questions.  I'm
sure they could also help figure out what the exact restrictions on the
formats are, if the documentation is not clear enough:

https://docs.aws.amazon.com/awssupport/latest/user/getting-started.html

>     Chris> Customizations, such as SSH credentials, would be specified
>     Chris> in a GuixSD operating system configuration file and built
>     Chris> into the VM image, so neither the EC2 metadata service, nor
>     Chris> hacks like the "cloud-init" script used by some distros,
>     Chris> would enter into the picture at all.
>
>     Chris> Some preliminary work in a similar spirit was already done in
>     Chris> the branch 'wip-deploy', but I don't think it was
>     Chris> EC2-specific in any way.  Perhaps by looking there, you can
>     Chris> find some inspiration?
>
> Here the immediate downside would be that stuff like auto-scaling does
> not work out of the box. Which some people consider one of the selling
> features of AWS, the prices for VM hosting being rather high.

I think auto-scaling is orthogonal to this discussion.  If we build a
way to deploy an EC2 instance from a GuixSD operating system
configuration file, then it implies that we also have the ability to
create an AMI from the operating system configuration file.  So
auto-scaling is definitely possible.  Whether or not auto-scaling works
for a given use case depends on the use case, not on the mechanism by
which the AMI used for auto-scaling was created.

>     Chris> I think it would be better to spend your energy on creating a
>     Chris> mechanism that allows an individual to build a GuixSD image
>     Chris> from their own operating system configuration file, import
>     Chris> that into EC2, and then launch an instance from it.  If such
>     Chris> a feature were available in GuixSD, you could do it once from
>     Chris> a desktop/laptop with a slow internet connection to create a
>     Chris> "control server" in the cloud (with a fast internet
>     Chris> connection), and then you could run it from the control
>     Chris> server as needed to quickly spin up whatever other instances
>     Chris> you might need.
>
> I think the above steps could be shortened somewhat and automated, if
> you know you're running on ec2.
>
> I don't see a way to cleanly import an image into AWS. This is however
> different for OpenStack, there you have an image service that does just
> what we need.

I haven't tried it myself, but I suspect that the "VM Import" feature is
the right approach.  I think it's probably the best thing to try first.
We would need to implement the mechanism for building a GuixSD system in
whatever VM format EC2 requires, and we would need to implement a Guile
interface for EC2 (or at least wrap the AWS CLI).

> I'll try my hand at optimizing these steps on the weekend.

Good luck!  If you need anything, I'm happy to talk.

Finally, I'd like to mention that although I work at AWS, this email
correspondence is entirely my own personal opinion and does not reflect
the views or opinions of my employer.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]