Terraform: Running terraform init on a subdirectory with s3 backend causes terraform workspace command to fail with uninitialized backend error

[sverchdotgov]

Terraform Version

$ terraform --version
Terraform v0.11.7
+ provider.aws v1.18.0

Terraform Configuration Files

provider "aws" {
  region = "us-east-1"
}

terraform {
  backend "s3" {}
}

resource "aws_vpc" "example" {
    cidr_block       = "10.0.0.0/16"
}

Expected Behavior

After running terraform init I can run terraform workspace commands against that remote s3 state, even if I run terraform init on a subdirectory.

Actual Behavior

When I run terraform init on a subdirectory terraform workspace things I haven't initialized the state.

Running terraform init pointing to the current directory:

$ pwd
/tmp/terraform-bug
$ rm -rf .terraform
$ terraform init -input=false -lock=true -lock-timeout=0s  -force-copy -backend=true -backend-config=bucket\=test-kitchen-repro -backend-config=key\=terraform.tfstate -backend-config=region\=us-east-1 -get=true -get-plugins=true  -verify-plugins=true /tmp/terraform-bug

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.18.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 1.18"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform workspace select kitchen-terraform-test-ubuntu

Workspace "kitchen-terraform-test-ubuntu" doesn't exist.

You can create this workspace with the "new" subcommand.

Running terraform init on a subdirectory:

$ rm -rf .terraform
$ mv main.tf module/
$ ls module/
main.tf
Joes-MacBook:terraform-bug sverch$ terraform init -input=false -lock=true -lock-timeout=0s  -force-copy -backend=true -backend-config=bucket\=test-kitchen-repro -backend-config=key\=terraform.tfstate -backend-config=region\=us-east-1 -get=true -get-plugins=true  -verify-plugins=true /tmp/terraform-bug/module

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.18.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 1.18"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Joes-MacBook:terraform-bug sverch$ terraform workspace select kitchen-terraform-test-ubuntu
Backend reinitialization required. Please run "terraform init".
Reason: Unsetting the previously set backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.

Failed to load backend: Initialization required. Please see the error message above.

Steps to Reproduce

See above. Create a basic terraform configuration and run terraform init on a subdirectory.

References

https://github.com/newcontext-oss/kitchen-terraform/issues/237

[jbardin]

Hi @sverchdotgov,

Thanks for filing this with the complete example.
Many commands still accept an argument for a configuration path in order to support legacy workflows, but that argument pattern does not work for commands which take positional arguments, like workspace, [un]taint, state *, etc.

For now the recommended workflow is to always run terraform from the root of the configuration for consistency, which also has the benefit of making it explicit which configuration the working directory has been initialized for.

We are planning to refactor the UI in an future release to improve the consistency of the various operations.

[sverchdotgov]

Thanks for the response @jbardin! So it sounds like running terraform against a directory that isn't the current directory is effectively not fully supported until you refactor the UI. Is that right? I think kitchen-terraform depends on that as a core part of how it works, so that's good to know explicitly. See https://github.com/newcontext-oss/kitchen-terraform/issues/237.

[ocervell]

This is a real issue when running automated builds with sub-folders.
Something like this in a Cloud Build pipeline would fail:

- name: gcr.io/{_PROJECT_NAME}/terraform:0.11.14
  args:
  - workspace
  - select
  - ${_WORKSPACE}

- name: gcr.io/{_PROJECT_NAME}/terraform:0.11.14
  args:
  - apply
  - rendered/{_COMPONENT}/terraform.tfplan
  - --autoapprove

[speller]

For now the recommended workflow is to always run terraform from the root of the configuration for consistency

Looks like a shift of the software consistency responsibility from developers to users. A documented bug is a feature (c). What's the problem to support setting the "working directory" from a command-line parameter?

[speller]

BTW, I don't understand what's the benefit of using the "per-directory" design pattern instead of the explicit setting of "entry points" and explicit file relations. Save 3 lines of code in a 5,000-line configuration and add 100 lines of workarounds in CI/CD scripts managing "correct" working directories of multiple TF configs and transforming relative file and dir paths to the "terraform current directory"?

[steerben]

This is a feature I am also missing - and I would really ask for consistency here in the cli as @speller already explained.
I think it would be a valuable feature, especially in CI/CD envs.

[isae]

Hi, @jbardin. It would be nice to have this behaviour documented somewhere in bold: basically it means that if you're initializing Terraform in a subdirectory, you have limited functionality in case of remote backend. I've just imported some resource and I can't unimport (state rm) it because of this problem, so I have to manually edit my state file in S3.

Also, if I try to explicitly define state file for state rm command (using -state flag), which must be ignored, according to documentation, my "local state" (pointing to remote one) is now broken: it is overwritten with empty local state.

My Terraform version is 0.12.28.

[jbardin]

For 0.14 we are planning to move to a global -chdir option to set the effective working directory as an alternative to the directory argument which cannot be supported for all commands.

https://github.com/hashicorp/terraform/pull/26087