Terraform: Unable to set block values in `terraform init -backend-config=...`

[lawliet89]

Terraform Version

Terraform v0.12.2
(independent of Terraform Providers)

Terraform Configuration Files

terraform {
  required_version = ">= 0.12"

  backend "remote" {
  }
}

# The rest are irrelevant

Expected Behavior

Consider the usage of the Terraform remote backend where configuration values are provided via the CLI.

Previously, in Terraform 0.11, I could do

terraform init -backend-config=organization=my-org \
  -backend-config=workspaces=[{name = "foobar"}]

I was able to set the values for the workspaces block for the remote backend via -backend-config CLI flag during terraform init.

Actual Behavior

Initializing the backend...

Error: Invalid backend configuration argument

The backend configuration argument "workspaces" given on the command line is
not expected for the selected backend type.

Steps to Reproduce

terraform init -backend-config=organization=my-org \
  -backend-config=workspaces=[{name = "foobar"}]

Additional Context

This change in behaviour is probably due to the commit which added validation against the Attributes of backend schemas only and does not bother with BlockTypes. As far as I could tell, there was no validation previously and the value was passed to the backend directly for deserialization.

This will affect https://github.com/gruntwork-io/terragrunt/issues/687.

[apparentlymart]

Thanks for reporting this, @lawliet89.

Indeed, before this worked as an accidental consequence of how Terraform was using HCL, and no longer works because workspace isn't a list of objects but rather a nested block.

I think we'll need to investigate a little deeper to see what is feasible as a response here. We might be able to address this by having the -backend-config handling check if the given name is of a block and specifically ask HCL to parse the argument as a list of objects in that case before converting it into the correct block representation. This means that we'd be bypassing some of the usual processing HCL does and so we'd likely have to implement some local validation for the contents of the objects in the list, but that should just be annoying rather than impossible.

In the meantime, passing this option via a file on disk would be the workaround, since an HCL file for backend config can use all of the HCL syntax features natively. However, I expect that generating a temporary file like that will not be convenient for terragrunt; generating a file on disk happens to how Terraform Cloud and Terraform Enterprise handle this situation, but they have more control over the execution environment than terragrunt tends to, given that terragrunt can be used directly as a CLI tool.

[jfpanisset]

Ran into this issue as well, I was able to use a disk file to specify the workspace, but spent time trying to get the workspaces=[{name=foo}] syntax working before I found this GitHub Issue. Until this can be resolved, would it make sense to add a note to the documentation at:

https://www.terraform.io/docs/backends/config.html#partial-configuration

to indicate that not being able to set the workspace name/prefix from the command line using -backend-config is a known limitation?

[pjaintaylor]

+1 for this, will definitely help with generating repository-based dynamic workspaces with Terraform Cloud!

[osulli]

Also encountered this for Terratest.

A documentation update would have been useful. Like @jfpanisset I also spent some hours trying to work this out.
I'll likely go down the route of specifying an HCL file.

[edmd]

We need to be able to pass variables on terraform init. This is critical.

We can't specify a file with sensitive information.