Guile AWS

Guile AWS

[Ricardo Wurmus]

Hi Guilers,

this is not quite a release announcement, but the code turns out to be
usable enough that I thought I’d share what I’ve got at this point.

Guile AWS is a library that lets you talk to Amazon Web Services such as
S3, the Elastic Compute Cloud (EC2), Elastic File System (EFS), Route53
(Amazon’s DNS server), etc.  The implementation is probably more
interesting than the library itself.

Guile AWS is little more than a language specification for the Guile
compiler tower, which compiles the JSON specification of the AWS APIs to
Scheme.  There’s also a bit of inelegant plumbing to actually make
requests to an API endpoint.

I’ve been hacking on this library on and off again, testing only those
bits that I needed, so I would be happy if others were to play with it
and/or improve it.

Here’s what a session might look like:

--8<---------------cut here---------------start------------->8---
(import (aws api elasticfilesystem-2015-02-01))

;; You can also use the parameters %aws-default-region, %aws-access-key,
;; and %aws-secret-access-key
(setenv "AWS_DEFAULT_REGION" "eu-central-1")
(setenv "AWS_SECRET_ACCESS_KEY" "…")
(setenv "AWS_ACCESS_KEY_ID" "AKIA…")

;; Create a file system with this unique creation token.
(CreateFileSystem
 #:CreationToken "my-guile-aws-filesystem"
 #:Tags (list (Tag #:Key "project" #:Value "guile-aws")
              (Tag #:Key "type" #:Value "test")))

;; => returns some JSON, including the fs-id …


(import (aws api ec2-2016-11-15))
(RunInstances #:MaxCount 1
              #:MinCount 1
              #:ImageId %my-ami
              #:KeyName %my-ssh-key-name
              #:SubnetId %my-subnet-id
              #:InstanceType type
              #:InstanceInitiatedShutdownBehavior "terminate"
              #:BlockDeviceMappings
              (list (BlockDeviceMapping
                     #:DeviceName "/dev/xvda"
                     #:Ebs
                     (EbsBlockDevice
                      #:DeleteOnTermination #true
                      #:VolumeType "gp3"
                      #:VolumeSize 2))
                    (BlockDeviceMapping
                     #:DeviceName "/dev/sdf"
                     #:Ebs
                     (EbsBlockDevice
                      #:DeleteOnTermination #true
                      #:SnapshotId "snapshot-whatever"
                      #:VolumeType "gp3"
                      #:Iops 6000)))
              #:TagSpecifications
              (list (TagSpecification
                     #:ResourceType "instance"
                     #:Tags (list (Tag #:Key "created-by"
                                       #:Value "guile-aws")
                                  (Tag #:Key "name"
                                       #:Value "guile-aws-test"))))
              #:SecurityGroupIds
              %my-security-groups
              #:UserData %encoded-userdata)

;; => returns some SXML response
--8<---------------cut here---------------end--------------->8---

The biggest flaw in Guile AWS is handling of responses.  Currently, the
responses are raw JSON or SXML (dependent on the API protocol).  I left
it this way because I found that converting the response to an opaque
record type just isn’t very nice; it made processing of responses quite
a bit harder.  If you have a good idea how to best represent the
responses in a more Schemey way, please do let me know!

Another open issue is plumbing.  A conversation with AWS APIs is
reminiscent of monadic computations (in that computations can fail at
any point and the flow should not be burdened with error handling), and
it also feels like prompts and delimited continuations would be a good
fit.  Currently, there is nothing here to simplify a conversation with
the API, e.g. to repeat API calls until a certain condition is met, to
time out requests or to abort polling after n retries, or even to just
handle errors.

The biggest obstacle was the variability in AWS APIs.  Some expect XML
payloads, others JSON, yet others expect a weirdly mangled request
string (some sort of custom stringy representation of the request
payload) — it’s a zoo!  I’m still pretty sure that there are a bunch of
ridiculous bugs in the code, and it’s difficult for me to come up with
good tests.  I’d appreciate any help!

Having said all that: this really works for my limited use cases.  I can
create EC2 instances, initialize them, tag them, terminate them, attach
volumes, create and configure EFS file systems, attach them, update DNS
records for newly generated machines, etc.

The code is here:

   https://git.elephly.net/gitweb.cgi?p=software/guile-aws.git

and here’s a Guix package definition:

--8<---------------cut here---------------start------------->8---
(use-modules ((guix licenses) #:prefix license:)
             (guix packages)
             (guix git-download)
             (guix utils)
             (guix build-system gnu)
             (gnu packages)
             (gnu packages autotools)
             (gnu packages databases)
             (gnu packages gnupg)
             (gnu packages guile)
             (gnu packages guile-xyz)
             (gnu packages pkg-config)
             (gnu packages web))

(define guile-aws
  (let ((commit "bed1a6624e0faac0830346e3bb1ac3581e8bb34b")
        (revision "20210409.1"))
    (package
      (name "guile-aws")
      (version (git-version "0" revision commit))
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://git.elephly.net/software/guile-aws.git")
                      (commit commit)))
                (file-name (git-file-name name version))
                (sha256
                 (base32
                  "129xxv4xq2pn1bax8gvq0h3p4bsfdflipv85bsq94pinks9jb0cd"))))
      (build-system gnu-build-system)
      (native-inputs
       `(("autoconf" ,autoconf)
         ("automake" ,automake)
         ("pkg-config" ,pkg-config)))
      (inputs
       `(("guile" ,guile-3.0-latest)))
      (propagated-inputs
       (let ((p (package-input-rewriting
                 `((,guile-3.0 . ,guile-3.0-latest))
                 #:deep? #false)))
         `(("guile-json" ,(p guile-json-3))
           ("guile-gcrypt" ,(p guile-gcrypt)))))
      (home-page "https://git.elephly.net/software/guile-aws.git")
      (synopsis "AWS client for Guile")
      (description "Guile AWS is pre-alpha software.  At the very
least it’s yet another demonstration that Guile’s compiler tower can
be used to generate an embedded domain specific language from JSON
specifications.")
      (license license:gpl3+))))
--8<---------------cut here---------------end--------------->8---

-- 
Ricardo

Re: [EXT] Guile AWS

[Thompson, David]

On Mon, Apr 12, 2021 at 11:48 AM Ricardo Wurmus <rekado@elephly.net> wrote:
>
> Hi Guilers,
>
> this is not quite a release announcement, but the code turns out to be
> usable enough that I thought I’d share what I’ve got at this point.
>
> Guile AWS is a library that lets you talk to Amazon Web Services such as
> S3, the Elastic Compute Cloud (EC2), Elastic File System (EFS), Route53
> (Amazon’s DNS server), etc.  The implementation is probably more
> interesting than the library itself.
>
> Guile AWS is little more than a language specification for the Guile
> compiler tower, which compiles the JSON specification of the AWS APIs to
> Scheme.  There’s also a bit of inelegant plumbing to actually make
> requests to an API endpoint.

Hell yeah, this rules!

Years ago I took the same approach to generate a Guile API for
CloudFormation (an unreleased experiment) and was hoping that someone
would do the same for the entire AWS API.

As far as error handling goes, the official AWS SDK for NodeJS may be
something to study. It uses asyncs (which is a syntax over simple
promises) to allow for threading together multiple API requests
together and handle errors without too much headache. Most of the time
I use the Ruby SDK, less often the Python SDK (boto3), and in both you
just have to catch exceptions or inspect responses yourself, so if
that's what you have to do with the guile-aws for the time being then
you're still on par with several official SDKs.  Those SDKs don't do
anything magical with the responses, either, so I consider being given
a big ol' deserialized json/xml response in the form of a compound
s-exp to be just fine.

- Dave

Re: [EXT] Guile AWS

[Ricardo Wurmus]

Thompson, David <dthompson2@worcester.edu> writes:

> Years ago I took the same approach to generate a Guile API for
> CloudFormation (an unreleased experiment) and was hoping that someone
> would do the same for the entire AWS API.

Your CloudFormation experiment was indeed the spark that ignited the
fire.  Guile AWS isn’t based on your code, but I learned a lot from
looking at it a few years back, and it planted the seed to implement
Guile AWS using the same approach.

> As far as error handling goes, the official AWS SDK for NodeJS may be
> something to study. It uses asyncs (which is a syntax over simple
> promises) to allow for threading together multiple API requests
> together and handle errors without too much headache. Most of the time
> I use the Ruby SDK, less often the Python SDK (boto3), and in both you
> just have to catch exceptions or inspect responses yourself, so if
> that's what you have to do with the guile-aws for the time being then
> you're still on par with several official SDKs.  Those SDKs don't do
> anything magical with the responses, either, so I consider being given
> a big ol' deserialized json/xml response in the form of a compound
> s-exp to be just fine.

Ah, that’s good to know.  Still, I think at the very least guile-aws
ought to present a unified data structure — either an alist, a JSON
expression, or some SXML, but not a different structure dependent on
whatever the API uses internally.

-- 
Ricardo