terraform init recursion problem

[spanktar]

Terraform Version

0.9.8

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "foo-tfstate-sandbox"
    key    = "us-east-1/devops/terraform.tfstate"
    region = "us-east-1"
  }
}

Expected Behavior

Backend should initialize

Actual Behavior

➜  SAND git:(master) ✗ pwd
~/GitHub/lithos/SAND

➜  SAND git:(master) ✗ terraform init -backend=true ..
Initializing configuration from: ".."...
Error copying source: lstat ~/Documents/GitHub/lithos/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/SAND/
SAND/README.md: file name too long

Steps to Reproduce

Our terraform directory has all of the .tf files in a single directory, and subdirectories for each built environment. In these subdirectories, there is a single terraform.tfvars file with env-specific variables in it. When trying to initialize the backend, we get this problem.

Take the following directory structure as an example:

```➜ lithos git:(master) ✗ lsd -R
drwxr-xr-x 2 spanky staff 68B Jun 9 14:08 DEV/
drwxr-xr-x 2 spanky staff 68B Jun 9 13:16 PROD/
drwxr-xr-x 3 spanky staff 102B Jun 13 09:53 SAND/
-rw-r--r-- 1 spanky staff 249B Jun 9 13:13 aws.tf
-rw-r--r-- 1 spanky staff 172B Jun 13 09:44 terraform.tf
-rw-r--r-- 1 spanky staff 6.6K Jun 13 09:38 variables.tf
-rw-r--r-- 1 spanky staff 90B Jun 9 13:13 version.tf

./SAND:
total 8
drwxr-xr-x 3 spanky staff 102B Jun 13 10:53 ./
drwxr-xr-x 31 spanky staff 1.0K Jun 13 09:27 ../
-rw-r--r-- 1 spanky staff 2.6K Jun 13 09:27 terraform.tfvars
```

The terraform.tfvars file sets up environment-specific values that are realized in variables.tf. Then we would have a different tfstate file in each ENV directory (SAND/DEV/PROD). The backend is defined in terraform.tf When we try to run the following, from within one of our ENV subdirectories we get the recursion error.

1. terraform init -backend=true ..

Important Factoids

We are not using terraform env yet, which is why we do things this way. In the past we've just used a simple shell script that runs terraform remote to set up our remote backends. With this new project we figured we'd try terraform init.

EDIT: Apparently remote is deprecated.

References

https://github.com/hashicorp/terraform/issues/518

[apparentlymart]

Hi @spanktar,

The terraform init command in 0.9 has a rather odd signature inherited from the old purpose of terraform init. This is going to change in 0.10, making what you did here work, but for 0.9 it's interpreting your first argument as a module to initialize, which is apparently causing the module loader to get rather confused given a module path of ...

Note that in 0.9 when you have the s3 backend enabled there is _no_ local terraform.tfstate file any longer, so you may find that the separate directory per environment is not actually needed. (There may be other reasons to still do that, which I can't see from your example here.)

Note that interpolations are not allowed in backend, so if you're not ready to use state environments (renamed to "workspaces" in 0.10) then Partial Configuration is the compromise approach, similar to terraform remote config.

To do this, you'd either pass the relevant settings directly on the command line or make a separate file for each environment and pass the appropriate filename.

[spanktar]

Thanks for this advice (re partial configurations), and that's exactly what I'm attempting to do right now. I'm trying to pass in variables to parameterize the backend in a separate file. Not to get into an unrelated support situation here :-) but I still get an error from the following command:

$ terraform init -backend=true -backend-config=backend.tfvars

Error loading backend config: 1 error(s) occurred:

* terraform.backend: configuration cannot contain interpolations

The backend configuration is loaded by Terraform extremely early, before
the core of Terraform can be initialized. This is necessary because the backend
dictates the behavior of that core. The core is what handles interpolation
processing. Because of this, interpolations cannot be used in backend
configuration.

If you'd like to parameterize backend configuration, we recommend using
partial configuration with the "-backend-config" flag to "terraform init".

Is there anywhere, a complete example of doing this with a file?

Related to above, the backend.tfvars file contains:

aws_account_name = "sandbox"
aws_region = "us-west-2"
environment = "devops"

and backend defined as:

terraform {
  backend "s3" {
    bucket = "tfstate-${var.aws_account_name}"
    key    = "${var.aws_region}/${var.environment}/terraform.tfstate"
    region = "${var.aws_region}"
  }
}

[spanktar]

OHHHHH, I get it.
The values in backend.tfvars are for the TERRAFORM {} block. Like so:

bucket = "FOO-tfstate-sandbox"
key = "us-west-2/devops/terraform.tfstate"
region = "us-west-2"

This is massively unclear. Some better documentation around init would be appreciated.

I look forward to 0.10 to see the other changes as well. Thanks for your time.

[spanktar]

Tested in 0.9.10 and still not working. Terraform files at root, terrafform.tfvars file in sandbox which is where we'd like to init the state into:

➜  sandbox git:(DEVOPS-826) ✗ terraform init \
-backend=true \
-backend-config=bucket=sandbox-us-east-1-tfstate \
-backend-config=key=/us-east-1/terraform.tfstate \
-backend-config=profile=sandbox \
-backend-config=region=us-east-1 \
-backend-config=role_arn=arn:aws:iam::1234567890:role/sandboxAccessRole ..
Initializing configuration from: ".."...

Error copying source: lstat /Users/spanky/Documents/GitHub/5G/lithos/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/resources/task-defs/backend/foo.json: file name too long

[apparentlymart]

The update for this is in 0.10-beta1, not in 0.9.10. It will not be included in any 0.9 release because it's a breaking change. It will be in 0.10 final when we get there, though.

[spanktar]

OK, thanks for the update. I thought 0.9.10 was out already, my bad. I'll keep this open to verify upon release.

[apparentlymart]

Oh, sorry I see where this might have been confusing... 0.9.10 and 0.10.0 are two different things; 0.9.10 was just a patch release, addressing one specific problem. 0.10.0 is the next major release and will include new features, including the change to make terraform init behave more like the other commands in how it processes its command line arguments.

[spanktar]

Just getting around to testing 0.10.x and it still isn't working as expected (by us).

We have a single directory of TF files, let's call that directory "terrafiles". We then have multiple subdirectories, one for each AWS account. In these directories is a single terraform.tfvars file that is used to apply different values for various environments. For example, we have sandbox, dev, and prod subdirectories. There are no .tf files in here.

We do not want to init such that the root is copied into these subdirs, we just want a state file in each. In the past (0.9x) we only had a slight issue where init created the .terraform state directory in the root (terrafiles) so after init, we manually moved it into the account subdirectory (terrafiles/sandbox/.terraform). Then we would run everything from within sandbox like so:

terraform plan ..

Now, in 0.10, this doesn't seem possible. For example:

➜  sandbox git:(master) ✗ /usr/local/bin/terraform init -backend=s3 -backend-config=bucket=sandbox-us-east-1-tfstate -backend-config=key=/us-east-1/spanky/terraform.tfstate -backend-config=profile=AWSPROFILE -backend-config=region=us-east-1 -backend-config=role_arn=arn:aws:iam::1234567890:role/sandboxAccessRole .
Initialized blank state with remote state enabled!
➜  sandbox git:(master) ✗ terraform plan ..
Backend reinitialization required. Please run "terraform init".

If we try to give a SOURCE and PATH to init, it tries to copy everything into sandbox, ending up in a recursion error again:

lstat /Users/spanky/Documents/GitHub/5G/orbit/lithos/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/sandbox/
sandbox/sandbox/resources/task-defs/backend/container.json: file name too long

I'm going to try the moving method again, but we'd hoped with the new init method taking a path, that this problem would not occur.

[spanktar]

UPDATE: init'ing in the main directory and manually moving the state directory down into sandbox works. (Output with >>> is from our script)

➜  terrafiles git:(master) ✗ /usr/local/Caskroom/terraform-0.10.2/0.10.2/terraform init -backend=true -backend-config=bucket=sandbox-us-east-1-tfstate -backend-config=key=us-east-1/spanky/terraform.tfstate -backend-config=profile=AWSPROFILE -backend-config=region=us-east-1 -backend-config=role_arn=arn:aws:iam::1234567890:role/sandboxAccessRole

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
 - Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "template" (0.1.1)...
- Downloading plugin for provider "aws" (0.1.4)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 0.1"
* provider.template: version = "~> 0.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
+ set +x

>>> Moving the state directory into './sandbox' ...

>>> Showing contents of '/Users/spanky/GitHub/terrafiles/sandbox' ...
total 656
drwxr-xr-x   5 spanky  staff   170B Aug 18 10:57 .
drwxr-xr-x  29 spanky  staff   986B Aug 18 10:57 ..
drwxr-xr-x   4 spanky  staff   136B Aug 18 10:57 .terraform
-rw-r--r--   1 spanky  staff   322K Aug 17 16:41 terraform.tfstate.backup
-rw-r--r--   1 spanky  staff   3.4K Aug 18 09:40 terraform.tfvars

>>> Showing contents of '/Users/spanky/GitHub/terrafiles/sandbox/.terraform' ...
total 8
drwxr-xr-x  4 spanky  staff   136B Aug 18 10:57 .
drwxr-xr-x  5 spanky  staff   170B Aug 18 10:57 ..
drwxr-xr-x  3 spanky  staff   102B Aug 18 10:57 plugins
-rw-r--r--  1 spanky  staff   652B Aug 18 10:57 terraform.tfstate

>>> Terraform initialization complete!

[VladRassokhin]

Here's another peculiar recursion issue easier to reproduce:
Given config

module "recursion" {
  source = "./"
}

terraform init won't stop on Ctrl+C (SIGTERM), have to send SIGKILL.
Terraform version v0.11.7

[hashibot]

Hello! :robot:

This issue relates to an older version of Terraform that is no longer in active development, and because the area of Terraform it relates to has changed significantly since the issue was opened we suspect that the issue is either fixed or that the circumstances around it have changed enough that we'd need an updated issue report in order to reproduce and address it.

If you're still seeing this or a similar issue in the latest version of Terraform, please do feel free to open a new bug report! Please be sure to include all of the information requested in the template, even if it might seem redundant with the information already shared in _this_ issue, because the internal details relating to this problem are likely to be different in the current version of Terraform.

Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.