Terraform: Remote state data source breaks with periods in key name

[Quanqued]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

[0]% terraform -v
Terraform v0.8.1
Also affects 0.8.4

Affected Resource(s)

• terraform_remote_state

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

data "terraform_remote_state" "foo" {
  backend = "s3"
  config {
    bucket = "s3-bucket"
    key = "tf/terraform.tfstate"
    region = "us-west-2"
  }
}

output "test" {
  value = "${data.terraform_remote_state.foo.test["foo.bar"]}"
}

s3://s3-bucket/tf/terraform.tfstate

{
    "version": 3,
    "terraform_version": "0.8.1",
    "serial": 5,
    "lineage": "329a5957-8dc8-442e-a3b9-a4d1c984a8d7",
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {
                "test": {
                    "sensitive": false,
                    "type": "map",
                    "value": {
                        "foo.bar": "xyz"
                    }
                }
            },
            "resources": {},
            "depends_on": []
        }
    ]
}

Debug Output

https://gist.github.com/Quanqued/b75e4b5dd4d964cc65e3b1bff12d7473

Panic Output

n/a

Expected Behavior

I should see the following output:

[0]% terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

test = xyz

Actual Behavior

What actually happened?
No output was generated

[0]% terraform apply
data.terraform_remote_state.foo: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Also, you can see that the map key was appended to the name of the output in the local state file.

"data.terraform_remote_state.foo": {
    "type": "terraform_remote_state",
    "depends_on": [],
    "primary": {
        "id": "2017-01-30 23:29:49.740093808 +0000 UTC",
        "attributes": {
            "backend": "s3",
            "config.%": "3",
            "config.bucket": "sf-ops",
            "config.key": "db/terraform.tfstate",
            "config.region": "us-west-2",
            "id": "2017-01-30 23:29:49.740093808 +0000 UTC",
            "test.%": "1",
            "test.foo.bar": "xyz"
        },
        "meta": {},
        "tainted": false
    },
    "deposed": [],
    "provider": ""
}

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Copy the tfstate file provided above to s3
2. Update the .tf file with the correct bucket location
3. Run terraform apply

Important Factoids

Are there anything atypical about your accounts that we should know?

The use case in particular where I ran into this is creating maps of domain names pointing to Route53 zone id's. It works fine within the local state, but when imported from the remote state you see this behavior.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here?

There was a similar issue specifically around AWS tagging.
https://github.com/hashicorp/terraform/issues/2143

[apparentlymart]

We don't seem to have a label for provider/terraform so I just added the "bug" label here, but this really ought to be tagged with the provider too.

[dpetzold]

With terraform 0.11.x I am getting a slightly different behavior. When a period is read in from a key element using terraform remote state it is interpreted as a dictionary. The output from the example above yields:

data.terraform_remote_state.foo: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

test = {
  foo = map[bar:xyz]
}

When I expected:

data.terraform_remote_state.foo: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

test = {
  foo.bar = xyz
}

[cehoffman]

I think the external provider bug https://github.com/terraform-providers/terraform-provider-external/issues/18 exists because of the same underlying bug.

[apparentlymart]

Hi all! Sorry for the long silence here.

First, I want to make explicit that my last comment above was from before joined the Terraform team, and so I was commenting as a outside contributor.

This issue is a tricky collision of a few different problems, but the root cause here is that the current provider protocol uses a special "flatmap" representation to pass nested collections to and from the provider, as we can see in the state shown in the original comment here. As a result, a map key containing a period is ambiguous in the current protocol.

This usually works in practice because Terraform knows statically that a particular attribute is a map, and so it knows that any periods after that point in the string must therefore be part of the map key. (because the provider type system allows only maps of strings, not maps of other maps)

The terraform_remote_state data source is using a strange hack to return dynamic attributes, which prevents Terraform from using the schema to properly decode the keys here. The current implementation of this data source attempts to emulate the expected "flatmap" structure directly itself, but apparently doesn't get this exactly right.

For Terraform 0.12 we are changing to no longer use this flatmap approach, and Terraform will instead save more natural data structures into the state and provider protocol:

"data.terraform_remote_state.foo": {
    "type": "terraform_remote_state",
    "depends_on": [],
    "primary": {
        "id": "2017-01-30 23:29:49.740093808 +0000 UTC",
        "attributes": {
            "backend": "s3",
            "config": {
                "bucket": "sf-ops",
                "key": "db/terraform.tfstate",
                "region": "us-west-2",
            },
            "id": "2017-01-30 23:29:49.740093808 +0000 UTC",
            "outputs": {
                "test": {
                    "foo.bar": "xyz"
                }
            }
        },
        "meta": {},
        "tainted": false
    },
    "deposed": [],
    "provider": ""
}

This new approach will remove this ambiguity, since we can easily see the difference between dots in a map key vs. an actual map.

The special "dynamic attributes" behavior of this particular data source is also problematic here, and so for 0.12 we intend to move the outputs themselves into a map called "outputs" (as shown in my state example above) which will then make this data source more "standard" and address several bugs specific to it. We're intending to automatically update references to this as part of the configuration upgrade tool that will come as part of Terraform 0.12, so the reference in the original comment here would become instead:

output "test" {
  value = "${data.terraform_remote_state.foo.outputs.test["foo.bar"]}"
}

Since the terraform provider is compiled into Terraform, it will get all of these fixes/changes as part of the 0.12 release. It does look like terraform-providers/terraform-provider-external#18 is a similar problem, but I expect a fix _there_ will require some adjustments within the provider itself (at the very least upgrading its provider SDK to the latest as of the 0.12 release, but possibly other work too) and so the fix for that one is likely to lag behind this one slightly.

[apparentlymart]

I just gave this a try with v0.12.0-alpha1.

First I generated a state file using the following "producer" configuration:

output "example" {
  value = {
    "foo.bar" = "baz"
  }
}

Then I read that output with terraform_remote_state in another "consumer" module (in a sibling directory in my case, for simplicity's sake):

data "terraform_remote_state" "example" {
  backend = "local"
  config = {
    path = "${path.module}/../a/terraform.tfstate"
  }
}

output "example" {
  value = data.terraform_remote_state.example.outputs.example["foo.bar"]
}

I applied the producer configuration first:

$ terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

example = {
  "foo.bar" = "baz"
}

...and then I applied the consumer configuration:

$ terraform apply
data.terraform_remote_state.example: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

example = baz

Success! :tada:

This works because the new state format makes use of the same new type system the rest of the configuration language uses from v0.12 onwards. The contents of the state file generated by this new build are a little different than we saw in the earlier comments in this issue:

{
  "version": 4,
  "terraform_version": "0.12.0",
  "serial": 1,
  "lineage": "1aca8f12-8e80-d585-07ae-fe0585f3ca8a",
  "outputs": {
    "example": {
      "value": {
        "foo.bar": "baz"
      },
      "type": [
        "object",
        {
          "foo.bar": "string"
        }
      ]
    }
  },
  "resources": []
}

The output type in this case is actually an object type rather than a map type, since we used an object literal { ... } directly rather than getting this value from an attribute that exports a map. However, if we force this value to be a map by passing it through a typed variable we can see how it looks in the more common case of having a map of strings:

variable "example" {
  type = map(string)
  default = {
    "foo.bar" = "baz"
  }
}

output "example" {
  value = var.example
}

{
  "version": 4,
  "terraform_version": "0.12.0",
  "serial": 2,
  "lineage": "1aca8f12-8e80-d585-07ae-fe0585f3ca8a",
  "outputs": {
    "example": {
      "value": {
        "foo.bar": "baz"
      },
      "type": [
        "map",
        "string"
      ]
    }
  },
  "resources": []
}

In both cases the state file now records the _exact_ type of the value, ensuring that when it is read in with terraform_remote_state it will behave exactly the same as it would had it been consumed by a calling module within a single configuration. Even in this case where it's a map like before, the recorded type is ["map", "string"] (the internal JSON serialization of map(string)) rather than just "map" as it was in prior versions.

The consumer's state also looks different in v0.12:

{
  "version": 4,
  "terraform_version": "0.12.0",
  "serial": 2,
  "lineage": "09b10950-a11e-3428-2108-411fbb9a08e3",
  "outputs": {
    "example": {
      "value": "baz",
      "type": "string"
    }
  },
  "resources": [
    {
      "mode": "data",
      "type": "terraform_remote_state",
      "name": "example",
      "provider": "provider.terraform",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "backend": "local",
            "config": {
              "value": {
                "path": "../a/terraform.tfstate"
              },
              "type": [
                "object",
                {
                  "path": "string"
                }
              ]
            },
            "defaults": null,
            "outputs": {
              "value": {
                "example": {
                  "foo.bar": "baz"
                }
              },
              "type": [
                "object",
                {
                  "example": [
                    "map",
                    "string"
                  ]
                }
              ]
            },
            "workspace": null
          }
        }
      ]
    }
  ]
}

Rather than flattening all of the attributes of terraform_remote_state down into a single JSON object, the new format uses more explicit JSON data structures that allow Terraform to recover the original types and values exactly. The outputs map in particular is serialized in an interesting way where the type and value are both recorded, which allows Terraform to remember whether the value had an object or map type even though both serialize as a JSON object.

So with all of that said, this issue seems to be fixed by the changes in v0.12. This is already available on an experimental basis in the alpha1 release, and will be included in the forthcoming v0.12.0 final release.

Thanks for reporting this!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.