Terraform: Cant get terraform init to work

[gtmtech]

Terraform 0.9.8

I have this terraform backend terraform.tf:

terraform {
    backend "s3" {
        region = "eu-west-1"
    }
}

I run this:

terraform init --backend-config=bucket=my-state-bucket --backend-config=key=terraform/terraform.tfstate

Initializing the backend...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your environment. If you forget, other
commands will detect it and remind you to do so if necessary.

I then run this:

terraform plan

Backend reinitialization required. Please run "terraform init".
Reason: Backend configuration changed for "s3"

Whats going wrong? This functionality doesnt seem to work as documented at all?

This is reproducible with a very minimal fileset

[gtmtech]

The problem is overriding the key with --backend-config=key= - this just doesnt work.

overriding bucket works, overriding role_arn works, overriding key yields this result.

Sadly without interpolation in the terraform backend config, and without being able to override the key on demand, terraform is unusable in its current form for handling multi-state single bucket scenarios :(

[jbardin]

@gtmtech,

Thanks for filing the issue. Can you explain what exactly you're doing with the key value? You should be able to parameterize the key value using the -backend-config flag like you've done, and it shouldn't be handled any differently from the bucket value.

We introduced named states in 0.9 with the "environments" feature (which will be transitioned to "workspaces" in 0.10) specifically to provide a solution to allow multiple states for a single config. Is there a particular reason that feature doesn't work for you?

[gtmtech]

@jbardin its completely bonkers, I cant get even the very basic working

I created this repository so you can see EXACTLY all my files

https://github.com/gtmtechltd/tfbug

the README contains the steps to reproduce.

[gtmtech]

(in the above example, i dont even use --backend-config , I just use the terraform.tf backend config which is pretty-much copy-pasted from the docs, and I still get the issue)

[gtmtech]

Debug for me for the above repository (using bucket geoff-test-bucket-3 as the brand new bucket):

~/tfbug$ terraform init
Initializing the backend...


Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your environment. If you forget, other
commands will detect it and remind you to do so if necessary.


~/tfbug$ terraform plan resources
Backend reinitialization required. Please run "terraform init".
Reason: Unsetting the previously set backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.

Failed to load backend: Initialization required. Please see the error message above.

Contents of brand new local .terraform/terraform.tfstate :

{
    "version": 3,
    "serial": 0,
    "lineage": "281c553a-757e-475c-922c-4b0a61cc7240",
    "backend": {
        "type": "s3",
        "config": {
            "bucket": "geoff-test-bucket-3",
            "key": "terraform.tfstate",
            "region": "eu-west-1"
        },
        "hash": 2915766998633377192
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

No terraform.state was uploaded into the bucket, the bucket is still empty.

[gtmtech]

"We introduced named states in 0.9 with the "environments" feature (which will be transitioned to "workspaces" in 0.10) specifically to provide a solution to allow multiple states for a single config. Is there a particular reason that feature doesn't work for you?"

^^^ this isnt even overriding key now - although yes I was doing it to have the concept of different environments - but I cant even get it working at all :-/

[gtmtech]

Ok - completely worked out now @jbardin

terraform init --backend-config doesnt work when specifying a directory e.g.

terraform init --backend-config=key=foo resources

(this is a logged open bug from March which isnt fixed yet >.< - ouch)

And terraform plan resources doesnt pick up the .terraform/terraform.state files that is a level above resources, in the case that the terraform backend is in the current directory (This is a broken regression from 0.8 - which always used to allow this).

The workaround unbelievably is to cd into the resources directory, run the terraform init, cd back out, copy the resources/.terraform directory into the parent .terraform directory, and THEN run terraform plan resources

uck.

Its almost as if terraform init and terraform plan look in different places for the local terraform.tfstate file

[gtmtech]

Confirmed workaround:

1. Put terraform.tf in your resources/ folder
2. cd into resources/ folder and run terraform init --backend-config=key=terraform.tfstate
3. cd into parent/ folder
4. mv resources/.terraform .terraform
5. run terraform plan

This is the only way this works. Otherwise its borked

[jbardin]

@gtmtech,

Ok, I see what's going on. You're initializing with the config in the current directory, but then providing a _different_ config for plan, which doesn't have the same backend configuration. If you had a matching backend config stanza in the target directory, it should work.

The init command, when provided with a config directory, copies the target config to the current directory and initializes it. You would then run other commands without specifying the directory.

This legacy behavior of init was rarely used, and surprising to most, so we opted to remove it in the 0.10 release. In 0.10 the init command will no longer copy the target config to the current directory, using it in-place, and should better align with the plan and apply commands. Note however if you're using multiple configuration directories, the backend stanza must still be identical between them or you will have to init again.

The .terraform/ directory should almost always be left as an implementation detail, and should not be moved relative to the working directory, otherwise module resource may fail.

[gtmtech]

That works - thanks a lot!!!! Its a bit weird to have a terraform.tf file duplicated all over the place, but I'll take it over having to move directories around any day 👍

[gtmtech]

"The init command, when provided with a config directory, copies the target config to the current directory and initializes it." - the trouble with this statement is you cannot init a config directory and pass --backend-config params as well. (there is another bug on this)

[jbardin]

True, and another reason we're removing the legacy init behavior altogether.

Glad you got it sorted out!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.