Terraform: Concating distinct list of complex object/map leads to wrong values

[bharathkkb]

Terraform Version

Terraform v0.12.29 or Terraform v0.13.1

Terraform Configuration Files

    foo = distinct(flatten([{
        test=""
        test2 = {
            test3 = "baz"
        }
    }]))
    bar = distinct(flatten([{
        test=""
        test2 = {
        }
    }]))
    all = concat(local.foo,local.bar)

Debug Output

Crash Output

Expected Behavior

I would have expected local.all to have value

[
  {
    "test" = ""
    "test2" = {
      "test3" = "baz"
    }
  },
  {
    "test" = ""
    "test3" = {}
  },
]

Actual Behavior

local.all has value

[
  {
    "test" = ""
    "test2" = {}
  },
  {
    "test" = ""
    "test2" = {}
  },
]

Steps to Reproduce

Additional Context

In particular this seems to happen with the usage of distinct. Using something like

    foo = flatten([{
        test=""
        test2 = {
            test3 = "baz"
        }
    }])
    bar = flatten([{
        test=""
        test2 = {
        }
    }])
    all = concat(local.foo,local.bar)

seems to work as intended. Maybe this is some complex type conversion behavior I missed 😅

References

[alisdair]

Thanks for reporting this! I'm able to reproduce it with Terraform 0.13.2 and the following reduced config:

locals {
  foo = distinct([ { a = "b" } ])
  bar = distinct([ { } ])
}

output "foo" {
  value = local.foo
}

output "bar" {
  value = local.bar
}

output "all" {
  value = concat(local.foo, local.bar)
}

Running terraform apply shows that the result of concat is incorrect:

Outputs:

all = [
  {},
  {},
]
bar = [
  {},
]
foo = [
  {
    "a" = "b"
  },
]

Removing the calls to distinct gives the expected result:

Outputs:

all = [
  {
    "a" = "b"
  },
  {},
]
bar = [
  {},
]
foo = [
  {
    "a" = "b"
  },
]

[apparentlymart]

I think this behavior is correct in the sense that the language rules are behaving as intended, though I do agree that it's confusing.

What's going on here, I believe, is that the distinct function is defined as taking a list of any element type and returning a list of that same element type. In the original example here, that means that the two initial locals then end up having the following types:

foo = list(object({
  test = string
  test2 = object({
    test3 = string
  })
}))
bar = list(object({
  test  = string
  test2 = object({})
}))

So far this is reasonable: the two values have distinct element types because their structure was different. However, things get confusing when we pass these results into concat, because it has a special case where if all of its arguments are lists and the list element types all have a common base type then it will produce a list of that common base type as the result. I think that type unification step ends up concluding that the common base type of the two element types is that of bar, because extraneous attributes are allowed under conversion between attributes. Therefore the result of concat ends up being this list type:

concat = list(object({
  test  = string
  test2 = object({})
}))

...but in this case we would, I think, ideally like it to have instead be:

concat = tuple([
  object({
    test = string
    test2 = object({
      test3 = string
    })
  }),
  object({
    test = string
    test2 = object({
    })
  }),
])

With all of that said, it seems like the main surprising behavior here is concat deciding to silently drop an object attribute in order to unify it with another object type. It would've been better for type unification to have failed here and thus allow concat to fall through to constructing the tuple type I showed above.

I don't think we can change the general type unification behavior now because that would potentially cause a successful result to be an error for other situations where the result is more defensible (though still not ideal, I think). However, perhaps there is some room for changing concat in particular either to reject a type unification result that loses object type attributes (which, I think, would be quite hard to achieve) or to just eschew the possibility of returning a list altogether and always return a tuple type.

Another option would be changing distinct to work with tuple types too, which would then make its behavior be more like flatten's. That would not address the root cause (concat would still have this odd behavior if its arguments are all lists for any other reason), but I think would address the specific situation reported here and may do so with fewer unintended consequences.

I find myself leaning slightly towards changing concat to never return a list, but I think this may warrant some research and experimentation to see if there may be unintended consequences of that.