driftctl - Support for Terraform Cloud
This contribution is a new feature.
Introduction
Project
You can find the driftctl project presentation here.
Context
In order to understand some parts of the contribution, you have to be familiar with Terraform Cloud.
What is Terraform Cloud/Terraform Enterprise?
Terraform Cloud is a managed service providing a consistent and reliable environment to manage Terraform runs.
Here is some screenshots of what the Terraform Cloud Dashboard looks like.
How Terraform Cloud Works?
- Write: Create new infrastructure or manage existing one that you’ve already written using Terraform
- Compose: Use Workspaces to manage your environments
- Plan: Create an execution plan
- Provision & Manage: Use Terraform Cloud’s run environment as an execution platform
- Collaborate & Share: Use the Private Module Registry to provide Self-Service Infrastructure
Here is a schema showing the Terraform Cloud architecture.
Terraform Enterprise focuses more on large enterprises by providing a self-hosted distribution of Terraform Cloud.
Current behavior
Currently, a user can retrieve the terraform.tfstate file from the following places:
tfstate://: Localtfstate+s3://: AWS S3tfstate+http://: HTTP Endpointtfstate+https://: HTTPS Endpoint
The idea is to bring support for Terraform Cloud/Terraform Enterprise.
Implement the solution
The logic is pretty straightforward as we can use the Terraform Cloud API to retrieve the current state for a given Workspace.
GET /workspaces/:workspace_id/current-state-version
Note that Terraform Cloud also retains historical state versions that we can retrieve using the following endpoint.
GET /state-versions/:state_version_id
Here is a sample request example to fetch the current state from the Workplace with the id ws-6fHMCom98SDXSQUv:
curl \
--header "Authorization: Bearer $TOKEN" \
--header "Content-Type: application/vnd.api+json" \
https://app.terraform.io/api/v2/workspaces/ws-6fHMCom98SDXSQUv/current-state-version
We will then receive a response with the following shape:
{
"data": {
"id": "sv-SDboVZC8TCxXEneJ",
"type": "state-versions",
"attributes": {
"vcs-commit-sha": null,
"vcs-commit-url": null,
"created-at": "2018-08-27T14:49:47.902Z",
"hosted-state-download-url": "https://archivist.terraform.io/v1/object/...",
"serial": 3
},
"relationships": {
"run": {
"data": {
"type": "runs"
}
},
"created-by": {
"data": {
"id": "api-org-hashicorp",
"type": "users"
},
"links": {
"related": "/api/v2/runs/sv-SDboVZC8TCxXEneJ/created-by"
}
},
"outputs": {
"data": [
{
"id": "wsout-J2zM24JPFbfc7bE5",
"type": "state-version-outputs"
}
]
}
},
"links": {
"self": "/api/v2/state-versions/sv-SDboVZC8TCxXEneJ"
}
}
}
The part we are interested in is the hosted-state-download-url attribute which provides a url from which we can download the raw state tfstate.
We can then use this url with the HTTPReader already present in driftctl which allows us to get a state from an https endpoint.
To summarize, here is the final workflow:
- Fetch hosted-state-download-url from the API with the provided
WORKSPACE_ID(tfstate+tfcloud://WORKSPACE_ID) and the API token through the providedtfc-token(--tfc-token TFC_TOKEN) - Use
HTTPReaderwith the retrievedhosted-state-download-url
Add the new IaC source
As said above, we will add a new IaC source to scan resources from the input Terraform statefile.
This new flag will be : tfstate+tfcloud://$WORKSPACE_ID with $WORKSPACE_ID representing the ID for the workspace whose current state version we want to fetch.
Define constants and Terraform Cloud types.
// Used in scan --from tfstate+tfcloud
const BackendKeyTFCloud = "tfcloud"
// Terraform Cloud API base root
const TFCloudAPI = "https://app.terraform.io/api/v2"
type TFCloudAttributes struct {
HostedStateDownloadUrl string `json:"hosted-state-download-url"`
}
type TFCloudData struct {
Attributes TFCloudAttributes `json:"attributes"`
}
// Body of the current-state-version response
type TFCloudBody struct {
Data TFCloudData `json:"data"`
}
Define our TFCloudReader method.
/*
workspaceId: retrieved from "tfstate+tfcloud://workspaceId"
opts: contains our token within the TFCloudToken attribute
*/
func NewTFCloudReader(client pkghttp.HTTPClient, workspaceId string, opts *Options) (*HTTPBackend, error) {
// 1. Fetch the current-state-version from the TFCloud API
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/workspaces/%s/current-state-version", TFCloudAPI, workspaceId), nil)
if err != nil {
return nil, err
}
// 2. Provide the right headers (with the token in Authorization)
req.Header.Add("Content-Type", "application/vnd.api+json")
req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", opts.TFCloudToken))
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return nil, err
}
/* 3. Test that we have the correct response
- We can have 401 in case the provided authentication api_token is wrong
- We can have 404 in case the provided workspace_id is wrong
*/
if res.StatusCode < 200 || res.StatusCode >= 400 {
return nil, errors.Errorf("error requesting terraform cloud backend state: status code: %d", res.StatusCode)
}
bodyBytes, _ := ioutil.ReadAll(res.Body)
body := TFCloudBody{}
err = json.Unmarshal(bodyBytes, &body)
if err != nil {
return nil, err
}
// 4. Retrieve the hosted-state-download-url from the body response
rawURL := body.Data.Attributes.HostedStateDownloadUrl
logrus.WithFields(logrus.Fields{"hosted-state-download-url": rawURL}).Trace("Terraform Cloud backend response")
opt := Options{}
// 5. Return a new HTTP reader with the hosted-state-download-url
return NewHTTPReader(&http.Client{}, rawURL, &opt)
}
The NewTFCloudReader function above will be triggered when we'll use tfstate+tfcloud.
This logic is defined in the main state backend file:
func GetBackend(config config.SupplierConfig, opts *Options) (Backend, error) {
backend := config.Backend
if !IsSupported(backend) {
return nil, errors.Errorf("Unsupported backend '%s'", backend)
}
switch backend {
// ""
case BackendKeyFile:
return NewFileReader(config.Path)
// "s3"
case BackendKeyS3:
return NewS3Reader(config.Path)
// "http"
case BackendKeyHTTP:
fallthrough
// "https"
case BackendKeyHTTPS:
return NewHTTPReader(&http.Client{}, fmt.Sprintf("%s://%s", config.Backend, config.Path), opts)
// "tfcloud"
// config.Path contains our workspace id and opts contains HTTP Headers and the API token
case BackendKeyTFCloud:
return NewTFCloudReader(&http.Client{}, config.Path, opts)
default:
return nil, errors.Errorf("Unsupported backend '%s'", backend)
}
Add some tests
To check that our code covers the different cases correctly, we will write three tests:
- Success to fetch URL with auth header
- Fail with wrong workspaceId
- Fail with bad authentication token
We will define an array of tests in which we will iterate.
Here is the example of the success test case when we manage to recover the state correctly.
{
// Name of the current test
name: "Should fetch URL with auth header",
// Refers to the NewTFCloudReader arguments
args: args{
workspaceId: "workspaceId",
options: &Options{
TFCloudToken: "TOKEN",
},
},
url: "https://app.terraform.io/api/v2/workspaces/workspaceId/current-state-version",
// Refers to the hosted-state-download-url result
wantURL: "https://archivist.terraform.io/v1/object/test",
wantErr: nil,
// Mock the the different http calls
mock: func() {
httpmock.Reset()
// Mock the Terraform Cloud API call
httpmock.RegisterResponder(
"GET",
"https://app.terraform.io/api/v2/workspaces/workspaceId/current-state-version",
httpmock.NewBytesResponder(
http.StatusOK,
[]byte(`
{
"data":{
"attributes":{
"hosted-state-download-url":"https:archivist.terraform.io/v1/object/test"
}
}
}`)
),
)
// Mock the state response from the hosted-state-download-url request
httpmock.RegisterResponder(
"GET",
"https://archivist.terraform.io/v1/object/test",
httpmock.NewBytesResponder(http.StatusOK, []byte(`{}`)),
)
},
},
Here is the main loop in which we check that each test matches what we expected.
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Mock the following HTTP calls
tt.mock()
// Call the reader with the args
got, err := NewTFCloudReader(&http.Client{}, tt.args.workspaceId, tt.args.options)
// Check if we wanted an error
if tt.wantErr != nil {
// Check the expected error with the actual one and return
assert.EqualError(t, err, tt.wantErr.Error())
return
} else {
assert.NoError(t, err)
}
assert.NotNil(t, got)
// Check the expected url with the actual one
assert.Equal(t, tt.wantURL, got.request.URL.String())
})
}
Final result
Retrieve your workspace ID and API token from your Terraform Cloud account.
We can now scan our resource with the command:
driftctl scan --from tfstate+tfcloud://$WORKSPACE_ID --tfc-token $API_TOKEN
Which in my case gives the following output telling me that 9 resources are not covered by IaC.
Scanned resources: (20)
Found resources not covered by IaC:
aws_iam_policy_attachment:
- role_test-arn:aws:iam::559417107340:policy/ConsoleMe
aws_iam_role:
- role_test
aws_iam_access_key:
- ******************** (User: test2)
- ******************** (User: admin)
aws_iam_policy:
- arn:aws:iam::559417107340:policy/ConsoleMe
- arn:aws:iam::559417107340:policy/test_policy_2
- arn:aws:iam::559417107340:policy/test_policy
aws_iam_user:
- admin
- test2
Found 9 resource(s)
- 0% coverage
- 0 covered by IaC
- 9 not covered by IaC
- 0 missing on cloud provider
- 0/0 changed outside of IaC
You can try the tool yourself by following the driftctl official documentation.
Takeaway
Problems encountered
The majority of the problems I encountered were related to Golang. It's not a language I am familiar with so I often had to go back and forth between my IDE and the docs.
What did I learn ?
This was my first contribution in Golang and also the first one using Terraform Cloud.