terraform should remember "configuration directory" after terraform init

[modax]

Hello,

I think it kind of makes sense for terraform to remember path to the "configuration directory" (i.e. ${path.root}) if it was specified with terraform init DIR. Then it would be possible to use terraform output and other commands which are currently unusable with such configuration. What is more, commands like plan, apply could then use it implicitly and it would be actually possible to pass plan to them as an argument. The path itself could be written to e.g. .terraform/root in the current directory and only be overwritten by another terraform init ANOTHER_DIR run which specified another configuration directory (or . to restore default).

Use case: I'm creating S3 buckets for terraform state files with a small terraform scripts on three AWS accounts. This terraform.tfstate with appropriate terraform.tfvars will be stored in git repo itself (since they will rarely change if ever) in three different branches. However terraform scripts are the same for each environment and even other deployments (i.e. globally reusable) and are stored elsewhere in central location. So in my opinion it makes much sense just to do terraform init /path/to/tf-s3-bucket-scripts/ and be able to use terraform as usual afterwards in that dir. For this particular case, I can live with only init, plan, apply working when I pass the path to each of them, but this is very inconvenient.

I know that in terraform init you write:

It is recommended to run Terraform with the current working directory set to the
root directory of the configuration, and omit the DIR argument.

but I believe this can be a first class citizen and serve people well!

[modax]

In addition, terraform could support environment variable (e.g. TF_PATH_ROOT in the spirit of ${path.root}) to select another configuration directory. This might be convenient for CI (e.g. throw away states in /tmp), not that convenient for manual executions (which "remembering" would be useful for).

[nbering]

I think that most people (myself included) using the config directory argument are doing such esoteric things with it that it would be hard to predict the results of such a change.

Personally I used it until recently for the exact opposite case. I didn't want .terraform directories littering my project's directory which contained dozens of root configurations, I had wrapped terraform in a script and had the script fill in the config path and state file paths so that they would all share the same .terraform directory.

In this scenario, remembering the config path could easily have caused other weird things to happen. Especially if two terraform commands were run in parallel, as the script ran init itself which may have caused some weird race conditions.

Fortunately, using the S3 backend has kind of forced me out of this weird pattern and I am much better for it. Now each root config has a .terraform directory instead of just having just one in the project root.

[apparentlymart]

Thanks for this suggestion @modax!

We're aware that right now the experience when the root configuration directory is not the current working directory has some wrinkles and holes, which is why we currently recommend against that. However, we're also aware that people are using it for various purposes e.g. with wrapper scripts, as @nbering described.

Given the large amount of changes around terraform init in recent versions our plan for the time being is to let things "soak" for a little while and see what workflows emerge, and then in a later release we'll settle on some better-defined behavior for this situation.

One thing we can say with some certainty is that the workflow of having the current working directory be the configuration root will be supported in the long term. What is more debatable is how exactly to deal with the situation where they are different. Since there are valid use-cases on both sides, we need to collect some more experience before we can make any decisions here.

But with that said, this issue is good feedback in this area, so thanks for opening it!

[ghost]

Any news about this issue? I also think an optional environment variable should be used, I tried with TF_DATA_DIR which is mentioned in the docs ( https://www.terraform.io/docs/configuration/environment-variables.html ) without luck.

[iuriaranda]

Our use-case for keeping the config directory separated from the working directory: we have some generic configurations that can be applied to different customers or setups, by just providing a custom tfvars file for each of them. We prefer to initialize terraform in the customer repo, as there's where their custom variables and backend config resides, keeping the Terraform configuration files outside the customer repository.

If we would initialize Terraform in the configuration directory, as you recommend, we would be limited to have just one state initialized at a time. Also there would be a risk of applying the wrong configuration to the wrong customer if we forget to reinitialize before switching setups.

Another inconvenient is that there are some Terraform cli commands that don't support a custom configuration directory, like state or output.

So, considering this, a solution like the proposed here would be really useful in this case.