Terraform: data "aws_ebs_volume" -- not finding/returning valid defined storage

[notjames]

Terraform Version

terraform 0.8.4

Affected Resource(s)

data "aws_ebs_volume"

Terraform Configuration Files

109 # aws --profile my_profile --region us-west-2 ec2 describe-volumes
110 #     --filters Name=tag:Name,Values=nvblock-enc-storage
111 #               Name=size,Values=10
112 #               Name=encrypted,Values=true
113 #               Name=volume-type,Values=gp2
114 data "aws_ebs_volume" "nonvolatile-block-storage"
115 {
116     count       = "${length(var.azones)}"
117
118     filter
119     {
120         name   = "tag:Name"
121         values = ["${var.environment}-nvblock-enc-storage"]
122     }
123
124     filter
125     {
126         name   = "size"
127         values = ["10"]
128     }
129
130     filter
131     {
132         name   = "encrypted"
133         values = ["true"]
134     }
135
136     filter
137     {
138         name   = "volume-type"
139         values = ["gp2"]
140     }
141
142     filter
143     {
144         name   = "availability-zone"
145         values = ["${element(var.azones, count.index)}"]
146     }
147 }

Debug Output

https://gist.github.com/notjames/b09d90bc0c02d43b0360d2a3dabf93ea
You'll notice that the paste there is an excerpt of what I believe are the relevant needs. If you need more, please let me know. I can give the whole file but prefer not to unless required.

Panic Output

N/A

Expected Behavior

There are three volumes that should have been found using the search filters in the data-resource request IE aws cli finds the volumes:

$ aws --profile cust --region us-west-2 ec2 describe-volumes \
     --filters Name=size,Values=10 \
                 Name=encrypted,Values=true \
                 Name=volume-type,Values=gp2 \
                 Name=tag:Name,Values=test-nvblock-enc-storage | \
    jq '.Volumes | length'

3

Actual Behavior

0 volumes are found.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create 1 ebs volume in a us-west-2 VPC in each AZ (a total of three volumes should be available).
2. Name each volume the same name; your choice.
3. Create an HCL file using the proper variables required and a data resource such as above with filters you would use to find those volumes.
4. terraform refresh or terraform apply or terraform plan fail for me finding 0 volumes.

Important Factoids

I've tried multiple permutations of filters, including removing all but one filter. I've also removed the count et al for testing. None of my tests resolve the issue of returning something. In short, data "aws_ebs_volume" is simply not working.

References

None that I could find.

[grubernaut]

Hi @notjames thanks for the issue!

I followed the reproduction steps as listed and created 3 identical EBS volumes, one for each us-west-2 availability zone. I then used the following TF configuration to filter the EBS volumes:

provider "aws" {
  region = "us-west-2"
}

variable "azones" { default = ["us-west-2a", "us-west-2b", "us-west-2c"] }

data "aws_ebs_volume" "ebs_volume" {
  count = "${length(var.azones)}"
  filter {
    name = "tag:Name"
    values = ["jake-test"]
  }
  filter {
    name = "volume-type"
    values = ["gp2"]
  }
  filter {
    name = "size"
    values = ["10"]
  }
  filter {
    name = "encrypted"
    values = ["true"]
  }
  filter {
    name = "availability-zone"
    values = ["${element(var.azones, count.index)}"]
  }
}

However, I was unable to reproduce the issue that you're seeing. The above configuration yielded me three separate aws_ebs_volume resources, one for each AZ.

I do believe you're seeing a legitimate error though, and would like to dive in further. If you write the data source without _any_ filters and inspect the debug output, does the raw AWS response match the same volumes you're seeing in the AWS console?

Eg:

data "aws_ebs_volume" "debugging" {
}

Should yield a debug log with the following line(s):

Found These Volumes ([]*ec2.Volume) () {
  <volume information>
}

[notjames]

I'm not sure what to use to answer the validation question you asked. The debug output finds 26 volumes, which is the correct number of volumes for this region. The volumes I'm seeking in the original request are also seemingly detected within the group.

Here's the output from the suggested debug code:
https://gist.github.com/notjames/bda9a2931263e8a58389c91381528d94

[grubernaut]

@notjames Interesting. It's definitely returning the full list of EBS Volumes for the region then?

I also see in the debug output that the correct error message is being shown that there were too many volumes returned, as expected.

When you initially tested removing all but one filter, which filter did you still include? Can we try only the size or volume_type filter? I would expect that to return more than one result, like the previous run we did without any filters.

[notjames]

Hrm. This is truly baffling. Now I'm not able to duplicate the issue. I'll keep digging to see what I did wrong. In the meantime, we can put this on hold pending probable resolution as segfault in user.exe. :)

[grubernaut]

That's why I initially wanted to get the debug output of the data lookup without any filters. When I first started to attempt to reproduce the issue, I wasn't getting any results back either, but found I was using the wrong credentials 🥇

[grubernaut]

@notjames Going to close this for now, happy to re-open if the issue occurs again. Thanks! <3

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.