terraform init can't deal with specified config path

[sj26]

We have an ops repo with packer/ and terraform/ subdirectories. We keep the terraform state in the repo root within .terraform/ and run our terraform commands from the root, but all terraform configuration is inside the terraform/ subdirectory. The reason is that these commands are usually run by automated tools from the source code root, and keeping the entry point in the repo root keeps it simple for new folks to on board, but we want to keep terraform configuration encapsulated.

This used to run fine in <= 0.8:

$ terraform remote config -backend=S3
  -backend-config=region=us-east-1
  -backend-config=bucket=ops.example.com
  -backend-config=key=terraform/terraform.tfstate

$ terraform plan -out .terraform/terraform.tfplan terraform/

$ terraform apply .terraform/terraform.tfplan

Now since 0.9+ instead of doing a remote config we do an init. We added a configuration file with the correct backend options. But the init command can't seem deal with configuration being in a sub-directory. Supplying SOURCE works fine but copies all files into current directory leaving a dirty git tree:

$ terraform init terraform/
Initializing configuration from: "terraform/"...
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your environment. If you forget, other
commands will detect it and remind you to do so if necessary.

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    new file:   terraform/backend.tf

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    files/ templates/ and.tf a.tf bunch.tf of.tf files.tf

$ git clean -fd

$ terraform plan terraform/

# everything back to normal...

Supplying SOURCE and PATH doesn't work, pretty expected:

$ terraform init terraform/ terraform/
Initializing configuration from: "terraform/"...
Error copying source: The destination path contains Terraform configuration files. The init command
with a SOURCE parameter can only be used on a directory without existing
Terraform files.

Please resolve this issue and try again.
exit status 1

We really just need to supply a PATH without a SOURCE and without requiring a fetch. Something like this might work:

$ terraform init - terraform/

Or the arguments could be flags instead of positional.

$ terraform init -path terraform/

[jbardin]

Hi @sj26,

I agree things can be a little confusing in this area after coming from the older remote setup.

Terraform really expects to be in the root of your configuration to find the data it needs. If terraform ran the initialization in the target directory, the .terraform directory would be created there, with saved backend settings and any modules. Those backend setting and modules then wouldn't be found in the current directory when you go to run plan or apply. While you can fool terraform by removing the copied files, any modules would still be missing.

What you can do for now is move just the backend config into its own .tf file in the top level directory, and add any "module" blocks that the other configs may need. You can then init without any arguments to setup the .terraform data, and targeting the subdirectories should then work.

I'll look into improving the UX around this, and see if we can relax the requirement for having the initialization data in the working directory.

[devth]

terraform init really needs symmetry with plan and apply.

Terraform really expects to be in the root of your configuration to find the data it needs

This is not consistent with plan and apply since they both accept [DIR-OR-PLAN] args. Many people and blog posts rely on this for complex multi-env and multi-project Terraform directory structures where files are not in the root dir. The workaround of adding module blocks doesn't work in these scenarios.

[rokka-n]

I agree with the previous comment. UX around init is really terrible and not only it inconsistent with plan/apply, but it also goes against the modular approach of building infrastructure and designing complex plans.

[sj26]

Thanks @jbardin! ☺️

[phamann]

Thank you for changing this behaviour @jbardin it was a big issue for us at Fastly. 😺

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.