Terraform: `terraform init` does not get providers from *all* workspaces' states

[mdavis40]

Hello!

I hope I've filled this template out properly. Please let me know otherwise and I'll correct any issues.

Terraform Version

Terraform v0.11.13

Terraform Configuration Files

backend.tf

terraform {
  backend "s3" {
    bucket               = "my-bucket"
    key                  = "my-state.tfstate"
    region               = "us-east-1"
    workspace_key_prefix = "terraform-workspace"
  }
}

initial main.tf

provider "aws" {
  region = "us-east-1"
}

provider "null" {}

resource "null_resource" "create-endpoint" {
  provisioner "local-exec" {
    command = "echo stuff"
  }
}

... whatever other resources are defined ...

updated main.tf

provider "aws" {
  region = "us-east-1"
}

... whatever other resources are defined ...

Debug Output

I did not include TF_LOG=trace as I believe all relevant information is available in this gist.

Crash Output

N/A

Expected Behavior

The terraform plan --out tfplan should execute successfully.

Actual Behavior

The terraform plan --out tfplan fails because the null provider was not downloaded upon initialization as the plugin is not known to be required at the point the terraform init is run.

You can resolve the issue by running terraform init a second time, after you have selected the proper workspace (and assuming the null_resource is present in your state). The second init run succeeds due to remote state being retrieved for the workspace, and terraform recognizes that the null provider is needed.

Steps to Reproduce

1. terraform init
2. terraform workspace new/select <workspace>
3. terraform plan --out tfplan
4. terraform apply tfplan
5. remove null_resource and null provider as seen in update main.tf block above
6. wipe out .terraform/ to simulate a run in CI job (or run your CI)
7. terraform init
8. terraform workspace select <workspace>
9. terraform plan --out tfplan

Following these steps will reproduce the error.

Additional Context

Able to reproduce in multiple situations (local workstation, CI).

Our relevant CI flow is as follows:

1. terraform init
2. terraform workspace select <name>
3. terraform plan
4. terraform apply

The .terraform/ is stashed between plan and apply stages, but is not persisted beyond a single run of the pipeline, thus terraform init is executed on each run of the pipeline. Perhaps this is our issue, although following the automation guidance, we seem to be adhering to best practices.

References

Could not locate any similar issues.

Thank you!
matt

[apparentlymart]

Hi @mdavis40! Sorry for this odd behavior, and thanks for reporting it.

The intended behavior is for terraform init to consult both the configuration and the remote state to decide which providers are required. In your case, it sounds like that mechanism hasn't worked correctly: you have a null_resource object in the state (which Terraform detects and tries to refresh during plan) but terraform init isn't detecting it for some reason.

If you still have the reproduction environment available, could you try running the command terraform providers just prior to the step where you would've run terraform init before? That command shows the provider usage tree that terraform init should be using to decide what to install, and I'm curious to see whether the null provider shows up in there. In your case, I expect the row for it to look something like this:

├── provider.null (from state)

A TF_LOG=trace run from the terraform init command in particular could provide valuable additional context too, since it may tell us whether init did indeed detect the null provider requirement and, if so, why it then didn't try to install it.

Thanks!

[mdavis40]

Thanks for responding.

Here’s a sample repo.

I’ve updated the previous gist as well with the information you requested.

The terraform init log is provided from step 3 here.

Hope it helps!

[apparentlymart]

Thanks for the extra details, @mdavis40!

The most interesting thing I see in that trace log is this curious 404 Not Found response when trying to fetch the state:

2019/04/26 18:09:12 [DEBUG] [aws-sdk-go] DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /tf-workspace-sample HTTP/1.1
Host: tf-sample-issue-reproduction-remote-state.s3.amazonaws.com
User-Agent: aws-sdk-go/1.14.31 (go1.11.5; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.13
Authorization: ***
X-Amz-Date: 20190427T010912Z
Accept-Encoding: gzip


-----------------------------------------------------
2019/04/26 18:09:12 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 27 Apr 2019 01:09:12 GMT
Server: AmazonS3
X-Amz-Id-2: **
X-Amz-Request-Id: **


-----------------------------------------------------
2019/04/26 18:09:12 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>tf-workspace-sample</Key><RequestId>86D252EC22B87C81</RequestId><HostId>EhsWOFtoQ9BSRPpBusDHqMiK6tBdqhqmCusRqMxllYFjbHnpXU5U/1L6N4IomAbLTG6vmyO5l2I=</HostId></Error>
2019/04/26 18:09:12 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/GetObject failed, not retrying, error NoSuchKey: The specified key does not exist.
    status code: 404, request id: 86D252EC22B87C81, host id: EhsWOFtoQ9BSRPpBusDHqMiK6tBdqhqmCusRqMxllYFjbHnpXU5U/1L6N4IomAbLTG6vmyO5l2I=

I'm not sure yet why this request failed, but it does make sense that the other parts of Terraform would consider this to mean that the state for this workspace doesn't exist at all yet and thus that no plugin requirements can come from there. It is strange that this would affect only terraform init and not the subsequent terraform plan or terraform apply step.

[mdavis40]

@apparentlymart thanks for your reply. Apologies for not including the log initially as that is crucial information.

To clarify, the terraform plan does indeed fail, if I do not select the workspace and re-run terraform init.

[apparentlymart]

Oh yes sorry I was imprecise in my closing statement there: terraform plan _is_ successfully detecting that the null provider is needed, which then returns an error because it is not installed. Whereas terraform init is failing to detect that the null provider is needed, which allows it to complete without an error. So the fail/success are inverted from what they appear to be, causing my notes about it to be confusing. :confounded:

The key question here, I think, is why terraform init is getting a 404 when it fetches the state but terraform plan is (presumably) not.

[mdavis40]

Do you have details on how terraform init should work with a workspace? Since you cannot switch to a specific workspace prior to running terraform init, it would be difficult to detect the required plugins in the case you've deleted a previously used provider based on what I have seen in the code itself (and the behavior described herein).

[apparentlymart]

Ahh yes, I think you've nailed the bug here: terraform init should be retrieving _all_ of the workspace states to decide what to install, not just the default workspace.

[mildwonkey]

Hi @mdavis40! I've updated the description of this issue now that you and @apparentlymart identified the root cause.

As a workaround for the time being, you could leave an empty provider block in your configuration until the resource is removed from all workspaces' states:

provider "null" { }

[mdavis40]

Thanks @mildwonkey ! Have worked around it already, just posted the issue out here for awareness/clarification if it was a bug or not.

Appreciate you, love using Terraform!

[mdavis40]

This issue appears to be resolved now, thank you!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.