From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ricardo Wurmus Newsgroups: gmane.lisp.guile.user Subject: Guile AWS Date: Mon, 12 Apr 2021 17:47:30 +0200 Message-ID: <87r1jfv71p.fsf@elephly.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="19287"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.15; emacs 27.2 To: guile-user@gnu.org Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Mon Apr 12 17:48:30 2021 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lVyne-0004sD-II for guile-user@m.gmane-mx.org; Mon, 12 Apr 2021 17:48:30 +0200 Original-Received: from localhost ([::1]:49720 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lVynd-0008T3-DR for guile-user@m.gmane-mx.org; Mon, 12 Apr 2021 11:48:29 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35310) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lVymz-0008R0-Ji for guile-user@gnu.org; Mon, 12 Apr 2021 11:47:49 -0400 Original-Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21119) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lVymv-0000CP-Rk for guile-user@gnu.org; Mon, 12 Apr 2021 11:47:49 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1618242456; cv=none; d=zohomail.com; s=zohoarc; b=FfVsxUhWh2JE4MWFY07AZGkHn0e3s4mOrjSaHuSy5jZwF7V9cSfnJnElaXcIeBn3mV1PNKpi2DmuDKIrFvCwIYrqswucdct9nJ8FZmRH4OA0hRmnU7pPTWV+SVEZd+F//XDZvILn2dVgT3x4iI2sTjob2PsvM6w9HeoV5rJU+xw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1618242456; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=INx5KXqM1kq5D2kvSWtS83yrqgv9Sakxvupfyw8jVU4=; b=e5NBRuOQSXBFTVqGQSCpIotNMcCOgiZDVsVqF9QbbDvZ49VlswI/axWisSRxr9mvOXuAKy8gazNp/v5DHFvJyUrThfiQ7szHvAqjFViqWej2Us0azP6Pc5a3LHVRlzTvpzLIl2kjol7gWeHhNwWG6D4mkh2celeg8HYveo8Fpck= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1618242455; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=INx5KXqM1kq5D2kvSWtS83yrqgv9Sakxvupfyw8jVU4=; b=OAeN4s4bOFcT0408R5pcm9dlByWUEuygrU0X0rl2AFTH639XWPmacU+3HQv3CgMz ic1eryPKu0AitET8rLOdfDWfHzMFY+57hjFiehHIfFDbRdY6uUo9ATrHwIUZtQZxzME wO55YM4d8ozOujWaUykDBNbc8g+LCGVny4eCZs5c= Original-Received: from localhost (p54ad4dc1.dip0.t-ipconnect.de [84.173.77.193]) by mx.zohomail.com with SMTPS id 1618242453603311.57046479490657; Mon, 12 Apr 2021 08:47:33 -0700 (PDT) X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:17410 Archived-At: Hi Guilers, this is not quite a release announcement, but the code turns out to be usable enough that I thought I=E2=80=99d share what I=E2=80=99ve got at thi= s point. Guile AWS is a library that lets you talk to Amazon Web Services such as S3, the Elastic Compute Cloud (EC2), Elastic File System (EFS), Route53 (Amazon=E2=80=99s DNS server), etc. The implementation is probably more interesting than the library itself. Guile AWS is little more than a language specification for the Guile compiler tower, which compiles the JSON specification of the AWS APIs to Scheme. There=E2=80=99s also a bit of inelegant plumbing to actually make requests to an API endpoint. I=E2=80=99ve been hacking on this library on and off again, testing only th= ose bits that I needed, so I would be happy if others were to play with it and/or improve it. Here=E2=80=99s what a session might look like: --8<---------------cut here---------------start------------->8--- (import (aws api elasticfilesystem-2015-02-01)) ;; You can also use the parameters %aws-default-region, %aws-access-key, ;; and %aws-secret-access-key (setenv "AWS_DEFAULT_REGION" "eu-central-1") (setenv "AWS_SECRET_ACCESS_KEY" "=E2=80=A6") (setenv "AWS_ACCESS_KEY_ID" "AKIA=E2=80=A6") ;; Create a file system with this unique creation token. (CreateFileSystem #:CreationToken "my-guile-aws-filesystem" #:Tags (list (Tag #:Key "project" #:Value "guile-aws") (Tag #:Key "type" #:Value "test"))) ;; =3D> returns some JSON, including the fs-id =E2=80=A6 (import (aws api ec2-2016-11-15)) (RunInstances #:MaxCount 1 #:MinCount 1 #:ImageId %my-ami #:KeyName %my-ssh-key-name #:SubnetId %my-subnet-id #:InstanceType type #:InstanceInitiatedShutdownBehavior "terminate" #:BlockDeviceMappings (list (BlockDeviceMapping #:DeviceName "/dev/xvda" #:Ebs (EbsBlockDevice #:DeleteOnTermination #true #:VolumeType "gp3" #:VolumeSize 2)) (BlockDeviceMapping #:DeviceName "/dev/sdf" #:Ebs (EbsBlockDevice #:DeleteOnTermination #true #:SnapshotId "snapshot-whatever" #:VolumeType "gp3" #:Iops 6000))) #:TagSpecifications (list (TagSpecification #:ResourceType "instance" #:Tags (list (Tag #:Key "created-by" #:Value "guile-aws") (Tag #:Key "name" #:Value "guile-aws-test")))) #:SecurityGroupIds %my-security-groups #:UserData %encoded-userdata) ;; =3D> returns some SXML response --8<---------------cut here---------------end--------------->8--- The biggest flaw in Guile AWS is handling of responses. Currently, the responses are raw JSON or SXML (dependent on the API protocol). I left it this way because I found that converting the response to an opaque record type just isn=E2=80=99t very nice; it made processing of responses q= uite a bit harder. If you have a good idea how to best represent the responses in a more Schemey way, please do let me know! Another open issue is plumbing. A conversation with AWS APIs is reminiscent of monadic computations (in that computations can fail at any point and the flow should not be burdened with error handling), and it also feels like prompts and delimited continuations would be a good fit. Currently, there is nothing here to simplify a conversation with the API, e.g. to repeat API calls until a certain condition is met, to time out requests or to abort polling after n retries, or even to just handle errors. The biggest obstacle was the variability in AWS APIs. Some expect XML payloads, others JSON, yet others expect a weirdly mangled request string (some sort of custom stringy representation of the request payload) =E2=80=94 it=E2=80=99s a zoo! I=E2=80=99m still pretty sure that = there are a bunch of ridiculous bugs in the code, and it=E2=80=99s difficult for me to come up w= ith good tests. I=E2=80=99d appreciate any help! Having said all that: this really works for my limited use cases. I can create EC2 instances, initialize them, tag them, terminate them, attach volumes, create and configure EFS file systems, attach them, update DNS records for newly generated machines, etc. The code is here: https://git.elephly.net/gitweb.cgi?p=3Dsoftware/guile-aws.git and here=E2=80=99s a Guix package definition: --8<---------------cut here---------------start------------->8--- (use-modules ((guix licenses) #:prefix license:) (guix packages) (guix git-download) (guix utils) (guix build-system gnu) (gnu packages) (gnu packages autotools) (gnu packages databases) (gnu packages gnupg) (gnu packages guile) (gnu packages guile-xyz) (gnu packages pkg-config) (gnu packages web)) (define guile-aws (let ((commit "bed1a6624e0faac0830346e3bb1ac3581e8bb34b") (revision "20210409.1")) (package (name "guile-aws") (version (git-version "0" revision commit)) (source (origin (method git-fetch) (uri (git-reference (url "https://git.elephly.net/software/guile-aws.git") (commit commit))) (file-name (git-file-name name version)) (sha256 (base32 "129xxv4xq2pn1bax8gvq0h3p4bsfdflipv85bsq94pinks9jb0cd")))) (build-system gnu-build-system) (native-inputs `(("autoconf" ,autoconf) ("automake" ,automake) ("pkg-config" ,pkg-config))) (inputs `(("guile" ,guile-3.0-latest))) (propagated-inputs (let ((p (package-input-rewriting `((,guile-3.0 . ,guile-3.0-latest)) #:deep? #false))) `(("guile-json" ,(p guile-json-3)) ("guile-gcrypt" ,(p guile-gcrypt))))) (home-page "https://git.elephly.net/software/guile-aws.git") (synopsis "AWS client for Guile") (description "Guile AWS is pre-alpha software. At the very least it=E2=80=99s yet another demonstration that Guile=E2=80=99s compiler = tower can be used to generate an embedded domain specific language from JSON specifications.") (license license:gpl3+)))) --8<---------------cut here---------------end--------------->8--- --=20 Ricardo