Terraform: Terraform not interpreting `compact` output as a List

[nathanielks]

Terraform Version

Terraform v0.11.3
+ provider.aws v1.10.0

Terraform Configuration Files

variable "region" {
  default = "us-east-1"
}

variable "name" {
  default = "tf-test-compact-list"
}

provider "aws" {
  region = "${var.region}"
}

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

variable "security_groups" {
  type    = "list"
  default = []
}

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"

  ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name = "allow_all"
  }
}

resource "aws_instance" "web" {
  ami           = "${data.aws_ami.ubuntu.id}"
  instance_type = "t2.micro"

  security_groups = "${compact(concat(list(aws_security_group.allow_all.id), var.security_groups))}"

  tags {
    Name = "${var.name}"
  }
}

Debug Output

https://gist.github.com/nathanielks/d8e6e579c3d0aa5a19f888b016cad5df

Crash Output

N/A

Expected Behavior

compact should produce a list that allows the resource to be created.

Actual Behavior

Terraform errored with Error: aws_instance.web: security_groups: should be a list.

Steps to Reproduce

terraform init

terraform plan

Additional Context

I also have a PR open at terraform-provider-aws, but I believe this is a bug in Terraform: https://github.com/terraform-providers/terraform-provider-aws/issues/3647

[javipolo]

We are experiencing the same issue with gcp provider, so I guess it is a terraform issue

[nathanielks]

I tested back to v0.7.9 and it doesn't work, so at least it's not a regression.

[apparentlymart]

Hi @nathanielks! Sorry for this annoying behavior.

The cause of this problem is that aws_security_group.allow_all.id is not yet known (it would show up as <computed> in the plan output), and the current language interpreter has a simplistic set of rules about unknown values: if you pass an unknown value to a function then the result is also unknown.

In this case this causes problems because <computed> is not a list and so it fails validation.

The good news is that this limitation is addressed by the improved configuration interpreter we're currently working on integrating. This is the current focus of the Terraform team and will be included in the next major release.

In the mean time, the usual workaround is to use the -target option to force Terraform to create the security group first, which then avoids the overly-conservative handling of unknown values because aws_security_group.allow_all.id will already be known by the time it is evaluated:

$ terraform apply -target=aws_security_group.allow_all
# ...
$ terraform apply
# ...

---

The new behavior for unknown values is improved in two ways in the new implementation that are relevant to this issue:

• Functions are able to now selectively accept and handle unknown values themselves, which means that list-oriented functions like these can (and will) be implemented to pass through unknown _elements_ without making the entire result unknown. In other words, the result of your expression there would be a list with a <computed> element inside, rather than the entire value being <computed>.

• Unknown values retain type information. In current Terraform <computed> is a type of its own, but in the new interpreter it is a special placeholder that _has_ an associated type. This means that a function can return "unknown list of strings" rather than just "unknown", and thus allow the type checker to do its work even though the _values_ aren't yet known.

[nathanielks]

hey, @apparentlymart! This is all good news! Thanks for laying it all out for me 😃

I went ahead and tested this out, but unfortunately it still doesn't work. Here's the log output for both commands: https://gist.github.com/nathanielks/3e7582405e38e181245e9489aff5b595

[apparentlymart]

Hi @nathanielks! I'm sorry that didn't work out as expected.

The log for the run with -target=aws_security_group.allow_all looks like it completed successfully, but I can't be sure since the log doesn't include the normal Terraform output where it would say Success! ... etc. Did it seem to be complete that command successfully?

Perhaps we can use terraform console to understand what Terraform is seeing here. For example, here's some experimenting I did with a null_resource to verify that Terraform was behaving the way I expected:

$ terraform console
> null_resource.test.id
8763957270277037288
> list(null_resource.test.id)
[
  8763957270277037288
]
> concat(list(null_resource.test.id), list())
[
  8763957270277037288
]
> compact(concat(list(null_resource.test.id), list()))
[
  8763957270277037288
]

If you do this with your aws_security_group.allow_all instead of my null_resource.test, do you get a similar result? (i.e. one that has a sg-nnnnnn-type id, rather than the timestamp seen here.)

[nathanielks]

@apparentlymart my apologies for not including the entire output! Yes, it completed as normal and expected by creating the security group.

And yes, within the console, compact produces a list as expected, that's what's so bizarre about the whole deal!

$ tf show
aws_security_group.allow_all:
  id = sg-b0b91a26
  description = Allow all inbound traffic
  egress.# = 0
  ingress.# = 1
  ingress.1403647648.cidr_blocks.# = 1
  ingress.1403647648.cidr_blocks.0 = 0.0.0.0/0
  ingress.1403647648.description =
  ingress.1403647648.from_port = 0
  ingress.1403647648.ipv6_cidr_blocks.# = 0
  ingress.1403647648.protocol = tcp
  ingress.1403647648.security_groups.# = 0
  ingress.1403647648.self = false
  ingress.1403647648.to_port = 65535
  name = allow_all
  owner_id = 577418818413
  revoke_rules_on_delete = false
  tags.% = 1
  tags.Name = allow_all
  vpc_id =

$ tf console
> compact(concat(list(aws_security_group.allow_all.id), var.security_groups))
[
  sg-b0b91a26
]

[apparentlymart]

Hi @nathanielks! Thanks for following up.

I'm starting to wonder if this is related to #17368, since this seems to be a misbehavior during the "validate" pass, and that's also where those other issues are failing. @jbardin is already looking at that one, so I'm going to wait until he's finished his investigation before digging further here.

Out of curiosity, did you ever have this configuration working on prior versions of Terraform? It looks like #17368 is a v0.11.3 regression, so if you've seen your configuration here working on earlier versions then that further suggests that these issues are related.

Thanks again for all the follow-up info here. Hopefully we can get to the bottom of this shortly.

[nathanielks]

@apparentlymart I'm happy to help in any way I can! Thanks for helping make my life easier with Terraform!

Unfortunately I just tried this new configuration, so I hadn't ever used this in the past. I did however try some regression testing and the error goes back to v0.7.9. I didn't test past that. The script if you're interested:

https://github.com/nathanielks/terraform-regression-tester

[apparentlymart]

Thanks for the regression testing, @nathanielks!

James managed to track down that other problem and indeed it seems to be unrelated to what you saw here, so we'll need to dig into this one separately. I've run out of time for today but when I'm able I'll see if I can reduce this to a minimal test case and then try to figure out what's going on here.

[nathanielks]

Sounds like a plan, @apparentlymart! If there's anything else I can do, please let me know. Happy to chase some rabbits if you don't have the bandwidth, I just need a direction! I'm still new at contributing to Terraform, so not entirely sure where exactly to look for things or how exactly to debug an issue like this.

[apparentlymart]

Hi @nathanielks,

We spent a little time looking at this today and ended up concluding that the behavior here is subtle/weird enough that it would probably be more efficient for us to postpone this for the moment and revisit once we've got further along with our current project of switching to the improved interpreter I mentioned before. It's likely that the new implementation will make the situation better here immediately, and so time we'd spend digging into the details here now would probably be throwaway work.

I'm sorry that this leaves you hanging for the moment. I'll follow up here again once we have the preview release of the new implementation ready (coming as soon as we have it integrated enough to be usable) and we can see to what extent the issue is fixed by that new implementation, and then plan for any additional fixes we might need to do in terms of that new implementation.

[nathanielks]

Sounds like a plan, @apparentlymart 👍

[tdmalone]

@nathanielks I might be missing something here, but if you put square brackets around the interpolation string, does Terraform correctly interpret it as a list?

eg.

security_groups = ["${compact(concat(list(aws_security_group.allow_all.id), var.security_groups))}"]

I had the same error with compacting a list of subnet IDs for a VPC, and the square brackets fixed it for me.

[nathanielks]

🤦‍♂️ 🤦‍♂️ 🤦‍♂️ 🤦‍♂️ 🤦‍♂️ 🤦‍♂️ 🤦‍♂️ 🤦‍♂️

@tdmalone thank you for the sanity check! Good eye... 👍

[sbuljac]

Why is this issue closed? Both compact and concat should return list, if you put it in square brackets you should get list inside the list.

[nathanielks]

@sbuljac I agree, and I think it's an idiosyncrasy to HCL they're looking to fix when HCL 2.0/Terraform 0.12 is officially released.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.