Terraform: Backend config not fully validated after input merge

[apparentlymart]

19250 has fixed a bug where the config for a backend wasn't being processed properly at all (likely related to #19249) but that methodology of doing a late merge means we don't get a chance to do a final validation of the merged config.

To complete this, it should behave similarly to the workflow used for _provider_ inputs:

• Decode the configuration using a derived schema where nothing is required (using NoneRequired), to get an initial raw config value that we can use to see what is set.
• Deal with the CLI args -backend-config and produce a partial cty.Value of those, and turn it into a "synthetic body" using configs.SynthBody
• Deal with input, prompting for any top-level required attributes that are not already set in either the raw config value or the CLI args value, producing another partial cty.Value of those, and another synthetic body.
• Use configs.MergeBody twice to first merge the real config body with the CLI values synth body, and then merge the resulting merged body again with the input values synth body.
• Finally, decode again with the original schema (with the required values still intact) to get the final config value with all of the other stuff merged in, or to get error messages if the config is invalid.

The body-merging strategy here is important to ensure that we can produce correct source location information for any error messages. configs.MergeBody ensures that any errors relating to values that were in the "real" body we started with will still show through with their correct source locations attached.

[dan-petty]

Is this the cause of the behavior I am seeing?

you used to be able to do:
terraform init -backend-config "bucket=yourbucket" -backend-config "region=region" -backend-config "key=statefile.tfstate" -backend-config "workspace_key_prefix=prefix"

with:

terraform {
  backend "s3" {
  }
}

As of 0.12.0-alpha2 terraform gives:
```Initializing the backend...

Error: Missing required attribute

on main.tf line 4, in terraform:
4: }

The attribute "key" is required, but no definition was found.

Error: Missing required attribute

on main.tf line 4, in terraform:
4: }

The attribute "bucket" is required, but no definition was found.

Error: Missing required attribute

on main.tf line 4, in terraform:
4: }

The attribute "region" is required, but no definition was found.```

[apparentlymart]

Hi @mixmatch!

The symptoms you are seeing there are the symptoms of the bug that was fixed in #19250, since that fix was merged after alpha2 was released. We intended the alpha builds primarily for _local_ testing of the new language features and bug fixes rather than use with real infrastructure, so we've not prioritized making a follow-up release with this backend fix.

It will definitely be included in the forthcoming beta1 release (along with the follow-on fix to _this_ issue) and possibly a subsequent alpha if there are sufficient new changes to warrant one for the purpose of verifying bug fixes and new configuration features.

For now, if you want to test _cautiously_ with a remote backend then you can do so by putting the backend configuration directly within the backend "s3" block for the duration of your testing.

[kofan]

Hi, when do you think this issue is going to be fixed?

[apparentlymart]

As I said in my last comment, it will be fixed before v0.12.0-beta1.

[jbardin]

I don't think there's anything remaining to do around the current backend implementation. The config already needs to be passed to the backend in order to load the defaults, which must validate the config as well. Fixing how config bodies are merged and decoded could possibly deal with future backend implementations that don't fully validate, but for now we would only be gaining an extra diagnostic with the line number of the backend block with the missing attribute[s].

Closing this out for now, since this will be taken care of by any new backend implementation which removes the dependency on the legacy SDK.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.