Terraform-provider-aws: support for data resources returning multiple entities?

[hashibot]

_This issue was originally opened by @chadgrant as hashicorp/terraform#10123. It was migrated here as part of the provider split. The original body of the issue is below._

---

I would think this would be handy / more succinct. Was this considered?

data "aws_subnet" "public" {
  tag {
    status = "public"
  }
}

resource "aws_alb" "test" {
  subnets = ["${data.aws_subnet.public.*.id}"]
}

[brownc]

This is a use case that I've been working through lately, in one example I'm looking for 2-3 Subnet IDs that our central admin team have configured for Lambdas to run in. Being tagged, they're easy to search - but for resilience practices I need more than one subnet.

The idea would be something like this:

# We need to find the Lambda Subnets for our App VPC
data "aws_subnet" "lambda_subnets" {
    filter {
        name   = "tag:Name"
        values = ["Lambda*"]
    }
}

# Lambda Function
resource "aws_lambda_function" "lambda_function" {
  <SNIP>
    vpc_config = {
        subnet_ids = ["${data.aws_subnet.lambda_subnets.id}"]
  <SNIP>
}

Right now I'm having to "hack" around this knowing that I have at least 2 subnets, and create a data source for Lambda_A, Lambda_B and so forth.

Would be great to wrap this up into one pattern and use tags as they're intended!

Thanks!

[vtorhonen]

I talked about this with someone from Hashicorp on HashiConf just recently. I promised to write a comment about it so here's my use case:

• Auto-scaling group consists of Nomad clients. Each node has a public IP associated with it.
• Each time a change occurs in the auto-scaling group (scaling up/down or a node is replaced) an event is triggered, which ultimately launches a parametrized Nomad job.
• Nomad job runs Terraform, which gathers list of public IP addresses in this certain ASG. These IP addresses are used to update a set of Route 53 records.

Currently this is not possible as the aws_instance data source does not support multiple entities. A simple example would look something like this:

data "aws_instance" "nomad_asg" {
  filter {
    name   = "tag:Name"
    values = ["my-awesome-asg"]
  }
  filter {
    name   = "tag:Datacenter"
    values = ["my-datacenter"]
  }
}

output "instances" {
  value = "${join(", ", data.aws_instance.nomad_asg.*.id)}"
}

If you run this the following error is printed out:

* data.aws_instance.nomad_asg: data.aws_instance.nomad_asg: Your query returned more than one result. Please try a more specific search criteria.

There's a workaround for this though. You can write a custom external datasource which uses AWS EC2 API to gather the same data. For example:

# Run "get-asg-ips" program. Returns list of IP addresses in JSON key "asg_ips"
data "external" "nomad_asg" {
  program = [ "get-asg-ips",
                "--asgname", "${var.asg_name}"
  ]
}

output "instance_ips" {
  value = "${join(", ", data.external.nomad_asg.result.asg_ips)}"
}

[bflad]

An update on this enhancement request: we have been accepting PRs for data sources that return multiple values that are separate from data sources that return a single value. As of v1.7.1 of the AWS provider, here are those data sources that return multiple values:

• aws_ami_ids
• aws_autoscaling_groups
• aws_availability_zones
• aws_ebs_snapshot_ids
• aws_instances
• aws_ip_ranges
• aws_subnet_ids

Since there is no real definition of "done" for this request here other than a data source that returns multiple subnet IDs (implemented as aws_subnet_ids) and a data source that returns multiple instance IDs (implemented as aws_instances), I'm going to close this issue. If there are other specific data sources you would like, please double check this repository for open issues/PRs (voting with 👍 reactions on the original comment) or open new issues. Thanks and happy Terraform'ing! 🎉

[thalesac]

Hi @bflad is there a feature request to imlement a aws_route_tables data source?

[soupdiver]

For me currently aws_security_groups would be interesting

Example:

resource "aws_security_group" "green" {
  name = "green"

  tags {
    Name = "green.cluster.dev"
  }
}

data "aws_security_group" "all_cluster" {
  tags = {
    Name = "*.cluster.dev"
  }
}

I'm working with kubernetes and kops and need to grant access to RDS instances from security groups created by kops

[bflad]

@soupdiver we currently support a "plural" data source for security groups (as of version 1.25.0 of the AWS provider): https://www.terraform.io/docs/providers/aws/d/security_groups.html

To encourage new issues for bugs/feature requests and reduce notifications for those following this issue, I'm going to lock this. 👍