Packaging Terraform, a Golang package

Packaging Terraform, a Golang package

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5415 bytes --]

Hi,

I'm trying to package Terraform, which is available under the Mozilla
Public License 2.0:

https://github.com/hashicorp/terraform

I'm not very familiar with how Golang programs are packaged, so I might
be missing some obvious things.  Please let me know if I am!

To begin, I thought I would first try to build it without creating a
package definition just yet.  Should be easy, right?

Clone the repo:

git clone https://github.com/hashicorp/terraform.git

Create a profile for hacking around that contains some tools that the
project seems to need:

guix package -i go make git bash grep findutils which -p .guix-profile

Make sure we're only using those dependencies:

eval "$(guix package --search-paths=exact --profile=$(realpath --no-symlinks .guix-profile))"

Enter the project directory:

cd terraform

Following the docs, run "make tools" to "install some required tools".
This downloads some stuff using "go get" (I'll probably have to put
those dependencies into the package definition later...):

--8<---------------cut here---------------start------------->8---
$ make tools
GO111MODULE=off go get -u golang.org/x/tools/cmd/stringer
GO111MODULE=off go get -u golang.org/x/tools/cmd/cover
GO111MODULE=off go get -u github.com/golang/mock/mockgen
--8<---------------cut here---------------end--------------->8---

Nice! Now, per the docs, we run make:

--8<---------------cut here---------------start------------->8---
$ make
==> Checking that code complies with gofmt requirements...
which: no stringer in (/home/marusich/terraform/.guix-profile/bin)
# We turn off modules for "go generate" because our downstream generate
# commands are not all ready to deal with Go modules yet, and this
# avoids downloading all of the deps that are in the vendor dir anyway.
GO111MODULE=off go generate ./...
2018/12/18 19:06:13 Generated command/internal_plugin_list.go
addrs/resource.go:256: running "stringer": exec: "stringer": executable file not found in $PATH
backend/operation_type.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
backend/local/hook_count_action.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
config/resource_mode.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
configs/provisioner.go:121: running "stringer": exec: "stringer": executable file not found in $PATH
configs/configschema/schema.go:86: running "stringer": exec: "stringer": executable file not found in $PATH
helper/schema/resource_data_get_source.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
plans/action.go:15: running "stringer": exec: "stringer": executable file not found in $PATH
plugin/mock_proto/generate.go:1: running "mockgen": exec: "mockgen": executable file not found in $PATH
states/instance_object.go:43: running "stringer": exec: "stringer": executable file not found in $PATH
states/statemgr/migrate.go:132: running "stringer": exec: "stringer": executable file not found in $PATH
terraform/context_graph_type.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
tfdiags/diagnostic.go:20: running "stringer": exec: "stringer": executable file not found in $PATH
make: *** [Makefile:80: generate] Error 1
[2] marusich@garuda.local:~/terraform/terraform
$ 
--8<---------------cut here---------------end--------------->8---

It seems like it failed to build because some tools were missing from
PATH, specifically "stringer" and mockgen".  I've tried various methods
of installing these things, but I can't seem to get it working.  How do
I get "stringer" and "mockgen"?

I ran these commands, but it didn't seem to work:

--8<---------------cut here---------------start------------->8---
[2] marusich@garuda.local:~/terraform/terraform
$ go get github.com/golang/mock/gomock
go: finding github.com/hashicorp/hcl2 v0.0.0-20181215005721-253da47fd604
go: finding github.com/terraform-providers/terraform-provider-aws v1.52.0
go: finding github.com/aws/aws-sdk-go v1.16.4
go: finding github.com/zclconf/go-cty v0.0.0-20181218225846-4fe1e489ee06
go: finding github.com/boombuler/barcode v1.0.0
go: finding github.com/pquerna/otp v1.0.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: finding github.com/golang/mock/gomock latest
go: downloading github.com/golang/mock v1.2.0
[0] marusich@garuda.local:~/terraform/terraform
$ go install github.com/golang/mock/mockgen
[0] marusich@garuda.local:~/terraform/terraform
$ go get -u -a golang.org/x/tools/cmd/stringer
go: finding golang.org/x/tools/cmd/stringer latest
go: finding golang.org/x/tools/cmd latest
go: finding golang.org/x/tools latest
go: downloading golang.org/x/tools v0.0.0-20181218204010-d4971274fe38
[0] marusich@garuda.local:~/terraform/terraform
$ go install golang.org/x/tools/cmd/stringer
[0] marusich@garuda.local:~/terraform/terraform
$ type -P mockgen
[1] marusich@garuda.local:~/terraform/terraform
$ type -P stringer
[1] marusich@garuda.local:~/terraform/terraform
$ 
--8<---------------cut here---------------end--------------->8---

I found this information about mockgen and stringer:

https://github.com/golang/mock

https://github.com/golang/go/issues/13874

Any help would be appreciated!  I'm not sure how to proceed.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Packaging Terraform, a Golang package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]

The answer is in the environment :)  Go uses a few variables:

- GOROOT, which points to the Go system installation.
- GOPATH, which points to the Go user "workspace", ~/go by default.

A GOPATH tree looks like this:

--8<---------------cut here---------------start------------->8---
> tree ~/go
~/go
├── bin
│   ├── foo
│   └── ipfs
└── src
    ├── github.com
    │   ├── aarzilli
    │   │   └── golua
    │   │       ├── example
    │   │       │   ├── alloc.go
    │   │       │   ├── basic.go
    │   │       │   ├── calls.lua
    │   │       │   ├── error.go
    │   │       │   ├── panic.go
    │   │       │   ├── quickstart.go
    │   │       │   └── userdata.go
    │   │       ├── LICENSE
    │   │       ├── lua
    │   │       │   ├── c-golua.c
    │   │       │   ├── golua.go
    │   │       │   ├── golua.h
    │   │       │   ├── lauxlib.go
    │   │       │   ├── lua
    │   │       │   │   ├── lauxlib.h
    │   │       │   │   ├── luaconf.h
    │   │       │   │   ├── lua.h
    │   │       │   │   └── lualib.h
    │   │       │   ├── lua_defs.go
    │   │       │   ├── lua.go
    │   │       │   └── lua_test.go
    │   │       ├── README.md
    │   │       └── TODO
--8<---------------cut here---------------end--------------->8---

It's populated automatically by =go get=, =go install=, etc.

The only thing it does not guess automatically is the path to ~/go/bin, which
you must add like all other executable paths to PATH.
For instance, from you ~/.profile or similar:

  export PATH=$PATH:~/go/bin

Hope that helps!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging Terraform, a Golang package

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

Hi Pierre,

Thank you for the info!  I did not know.  I will use this information to
try hacking around some more.  Hopefully it will work now!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Packaging Terraform, a Golang package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 147 bytes --]

You are welcome!

Also note that you need not set GOROOT, and GOPATH defaults to ~/go if unset.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Packaging Terraform, a Golang package

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]


Chris Marusich <cmmarusich@gmail.com> writes:

> Hi,
>
> I'm trying to package Terraform, which is available under the Mozilla
> Public License 2.0:

I have played with packaging Terraform previously, there are some
patches here [1]. I've also got a few more Terraform related packages in
this branch here [2].

1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30806
2: https://git.cbaines.net/guix/log/?h=terraform

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Packaging Terraform, a Golang package

[Chris Marusich]

[-- Attachment #1.1: Type: text/plain, Size: 3962 bytes --]

Hello Chris and Pierre!

Thank you for your tips regarding Golang and Terraform.  I've built it -
both ad-hoc and using a slightly modified version of Chris' original
package definition.  I'm still getting used to Golang's build system and
our own golang-build-system, so please bear with me!

I was able to run the official Terraform build steps on the latest
Terraform release by issuing the following commands - but I'm not sure
if this actually produced the terraform program:

mkdir terraform
cd terraform/
wget https://github.com/hashicorp/terraform/archive/v0.11.11.tar.gz
tar -xf v0.11.11.tar.gz
mkdir -p src/github.com/hashicorp
cp -r terraform-0.11.11 src/github.com/hashicorp/terraform
guix package -i go make git bash grep findutils which coreutils diffutils -p .guix-profile
mkdir gobin
eval "$(guix package --search-paths=exact --profile=$(realpath --no-symlinks .guix-profile))"
export GOPATH="$(pwd)"
export GOBIN="$GOPATH/bin"
export PATH="$PATH:$GOBIN"
cd src/github.com/hashicorp/terraform/
make tools
make

Those last two commands are the ones listed in Terraform's README.md.
It looks like the final command, "make" will execute the "fmtcheck",
"generate", and "test" recipes in that order.  Here are the relevant
parts of the Makefile:

--8<---------------cut here---------------start------------->8---
default: test

[...]

# test runs the unit tests
# we run this one package at a time here because running the entire suite in
# one command creates memory usage issues when running in Travis-CI.
test: fmtcheck generate
	go list $(TEST) | xargs -t -n4 go test $(TESTARGS) -timeout=2m -parallel=4

[...]

# generate runs `go generate` to build the dynamically generated
# source files.
generate:
	@which stringer > /dev/null; if [ $$? -ne 0 ]; then \
	  go get -u golang.org/x/tools/cmd/stringer; \
	fi
	go generate ./...
	@go fmt command/internal_plugin_list.go > /dev/null

[...]

fmtcheck:
	@sh -c "'$(CURDIR)/scripts/gofmtcheck.sh'"
--8<---------------cut here---------------end--------------->8---

So, in essence this runs "go generate", "go fmt", and "go test".  The
script gofmtcheck.sh seems to be a read-only linter to assist the
Terraform maintainers in finding badly formatted Golang files.

This is slightly different from the installation procedure that our
go-build-system runs.  When you build the attached package (which
packages an older version of terraform for now), our go-build-system
runs steps like the following (in order from top to bottom):

* Phase: unpack: Make a "src/github.com/hashicorp/terraform" directory.
* Phase: setup-environment: set GOPATH to (getcwd) and GOBIN to $out/bin.
* Phase: build: go install -v -x '-ldflags=-s -w' github.com/hashicorp/terraform
* Phase: check: go test github.com/hashicorp/terraform

All in all, this has raised some questions in my mind:

1) Is it bad that our package definition isn't running "go generate" or
"go fmt"?  Do you know if "go install" does this for us somehow?  I
don't think "stringer" or "mockgen" are present in the build
environment, but our "go install" invocation seems to build terraform
even without them.  I wonder if the built terraform is broken in some
ways because we didn't run the code generation/formatting steps.

2) After I ran "make" ad-hoc, I couldn't find a built "terraform"
executable anywhere.  Where is it?  Am missing something obvious, or
could it be that the official documentation incomplete and I need to ask
upstream for advice?

3) The official instructions seem to arbitrarily choose to run the build
in parallel, using 4 threads, which means this package won't play nice
with build arguments like --cores.  I suppose I might need to work with
upstream to fix that.

This feels so close, and yet so far.  Hopefully we only have a little
more to do to get Terraform packaged well!  Thank you again for your
help.

-- 
Chris

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Add-terraform.patch --]
[-- Type: text/x-patch, Size: 2905 bytes --]

From 71d654e2738c6ef870441d4234632bd30e93c74a Mon Sep 17 00:00:00 2001
From: Chris Marusich <cmmarusich@gmail.com>
Date: Thu, 20 Dec 2018 01:24:59 -0800
Subject: [PATCH] gnu: Add terraform.

This patch is slightly modified from Christopher Baines' original patch to
accommodate some changes that have taken place recently in the Guix source
tree.  But it's essentially the same.  It is still probably not suitable for
inclusion in Guix just yet.

* gnu/packages/terraform.scm (terraform): New variable.
---
 gnu/packages/terraform.scm | 40 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/gnu/packages/terraform.scm b/gnu/packages/terraform.scm
index f14b152fd..de2288dda 100644
--- a/gnu/packages/terraform.scm
+++ b/gnu/packages/terraform.scm
@@ -47,3 +47,43 @@ the inputs and outputs for modules of the Terraform infrastructure management
 tool.  These can be shown, or written to a file in JSON or Markdown formats.")
     (home-page "https://github.com/segmentio/terraform-docs")
     (license license:expat)))
+
+(define-public terraform
+  (package
+    (name "terraform")
+    (version "0.11.3")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/hashicorp/terraform")
+                    (commit (string-append "v" version))))
+              (sha256
+               (base32
+                "0637x7jcm62pdnivmh4rggly6dmlvdh3jpsd1z4vba15gbm203nz"))))
+    (build-system go-build-system)
+    (arguments
+     '(#:import-path "github.com/hashicorp/terraform"
+       #:phases
+       (modify-phases %standard-phases
+         ;; I'm not sure what purpose they serve, but they are readonly, so
+         ;; they break the reset-gzip-timestamps phase.
+         (add-after 'install 'delete-test-fixtures
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; If delete-file-recursively fails, it won't throw an exception.
+             ;; However, if it doesn't do its job, the build will fail, so
+             ;; we'll know one way or another.
+             (delete-file-recursively
+              (string-append
+               (assoc-ref outputs "out")
+               "/src/github.com/hashicorp/terraform/config/module/test-fixtures"))
+             #t)))))
+    (synopsis "Tool for building and changing computing infrastructure")
+    (description
+     "Terraform uses descriptions of infrastructure written in @acronym{HCL,
+Hashicorp Configuration Language} which describe graphs of resources,
+including information about dependencies. From this, Terraform can plan and
+apply changes to the described resources.
+
+Terraform uses plugins that provide intergrations to different providers.")
+    (home-page "https://www.terraform.io/")
+    (license license:mpl2.0)))
-- 
2.20.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Packaging Terraform, a Golang package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]


> 1) Is it bad that our package definition isn't running "go generate" or
> "go fmt"?  Do you know if "go install" does this for us somehow?  I
> don't think "stringer" or "mockgen" are present in the build

"go generate" is used to generate code, it's like a preprocessor.  Most Go
programs don't use it, but if they do, then it's needed.

"go fmt" should not be part of the build process in my opinion.  A code
formatter fits better into a Git hook or something.

> 2) After I ran "make" ad-hoc, I couldn't find a built "terraform"
> executable anywhere.  Where is it?  Am missing something obvious, or
> could it be that the official documentation incomplete and I need to ask
> upstream for advice?

I haven't looked into it, but it should be either in ~/go/bin or in the package
source folder, something like ~/go/src/.../terraform/.

> 3) The official instructions seem to arbitrarily choose to run the build
> in parallel, using 4 threads, which means this package won't play nice
> with build arguments like --cores.  I suppose I might need to work with
> upstream to fix that.

I think Go decides this on its own.  Off the top of my head, there is an
environment variable to control the number of CPU threads used globally,
GO_xxx_CORES or something like that.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]