Terraform: Terraform 0.11.9 does not take in account any more the .auto.tfvars

[ophintor]

Terraform Version

Terraform v0.11.9
+ provider.archive v1.1.0
+ provider.aws v1.41.0
+ provider.null v1.0.0
+ provider.random v2.0.0
+ provider.template v1.0.0
+ provider.tls v1.2.0

Expected Behavior

I expected terraform to populate some of my vars with the values declared in the .auto.tfvars files.

Actual Behavior

When running a terraform plan/apply, the values of some variables declared in a *.auto.tfvars file are not taken in account any more and terraform asks for them during runtime - breaking my automated pipeline.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
var.allow_ssh_cidr_blocks
  Whitelist for SSH access to EC2 instances.

  Enter a value:

$ cat ip-whitelists.auto.tfvars
allow_ssh_cidr_blocks = [
  "x.x.x.x/32", 
  "x.x.x.x/32", 
  "x.x.x.x/32", 
  "x.x.x.x/32", 
  "x.x.x.x/32", 
]

Steps to Reproduce

terraform apply

Additional Context

I have double checked by downgrading. Terraform 0.11.8 works.

[lorengordon]

Seconded. I'm also running into this!

[jparten]

I will add this also seems to be affecting loading of "terraform.tfvars". Not just "*.auto.tfvars".

[svanharmelen]

@ophintor I'm trying to reproduce this issue, but without any luck. I tried using a simple config like this:

$ cat main.tf
variable "allow_ssh_cidr_blocks" {
  type = "list"
}

resource "null_resource" "hello" {
  triggers {
    cidr = "${var.allow_ssh_cidr_blocks[2]}"
  }
}

And a *.auto.tfvars like this:

$ cat ip-whitelists.auto.tfvars
allow_ssh_cidr_blocks = [
  "192.168.0.1/32",
  "192.168.0.2/32",
  "192.168.0.3/32",
]

So could you maybe share a config that hits this problem so I can have a closer look at what is happening?

Thanks!

[arichiardi]

I tried to reproduced as well but it is only happening in some particular condition which I cannot for the life of me figure out. This basically disrupts our whole workflow.

[lorengordon]

Maybe try it with a terraform.tfvars file, _and_ a *.auto.tfvars file? I'm typically using terragrunt as a wrapper, and so I have both in the directory, though the terraform.tfvars file generally has only the terragrunt variable (used by terragrunt, not the actual terraform root module). In addition, I'll typically pass a couple -var-file arguments on the command line.

[arichiardi]

I use a wrapper as well, but there are some things handy to have in an .auto file like secrets.

[Tony-Proum]

Hello,
I run into this issue using Terraform Enterprise:
New workspaces created using terraform 11.9 does'nt take care of the *.auto.tfvars files.
Old workspaces created with terraform 11.8 does
As I use interpolated variables and static files, my workspace should contains a terraform.tfvars and a *.auto.tfvars . I'm agree with @jparten seems related to the presence of both terraform.tfvars and *.auto.tfvars in the same workspace

[ophintor]

So here is a simpler example. The way I'm running it in my real environment I also add variables using -var, which seems to be the cause of this issue. What I'm not sure is whether the fact that I'm using -var is preventing me from using *.auto.tfvars files? I believe this didn't happen in the previous version?

My files:

$ cat main.tf
variable "allow_ssh_cidr_blocks" {
  type = "list"
}

resource "null_resource" "hello" {
  triggers {
    cidr = "${var.allow_ssh_cidr_blocks[2]}"
  }
}
$ cat ip-whitelists.auto.tfvars
allow_ssh_cidr_blocks = [
  "192.168.0.1/32",
  "192.168.0.2/32",
  "192.168.0.3/32",
]

Run a simple terraform apply:

$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + null_resource.hello
      id:            <computed>
      triggers.%:    "1"
      triggers.cidr: "192.168.0.3/32"


Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.hello: Creating...
  triggers.%:    "" => "1"
  triggers.cidr: "" => "192.168.0.3/32"
null_resource.hello: Creation complete after 0s (ID: 1567257724528703682)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Run it again just to double check:

$ terraform apply
null_resource.hello: Refreshing state... (ID: 1567257724528703682)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Now run it with an extra var:

$ terraform apply -var=abc=123
var.allow_ssh_cidr_blocks
  Enter a value: 

PS: in my case I don't have a terraform.tfvars and I'm using different workspaces, although not in this particular case, so not sure if workspaces are relevant here.

[mr-bo-jangles]

I believe this didn't happen in the previous version?

It didn't happen previously, we've just had our build pipeline break because of this with the following command line terraform apply -var app_version=${CIRCLE_SHA1} -auto-approve -input=false Our pipeline expects the automatic loading of terraform.tfvars. We're currently testing if specifying it manually resolves the issue as a workaround for now.

[svanharmelen]

I was able to reproduce this problem now and have a fix for it. The problem occurs when using variables files (terraform.tfvars and/or *auto.tfvars) in combination with command line variables (e.g. -var or -var-file).

As soon as my colleagues in the US wake up I'll discuss this bug to see if we can cut a bugfix release to fix this problem.

[mr-bo-jangles]

I can confirm that specifying the files by name with -var-file works as a workaround for now

[svanharmelen]

Terraform v0.11.10 is just released to fix this problem. Sorry for any inconvenience caused!

[lorengordon]

@svanharmelen thanks! though, getting access denied when downloading the new release. is that just a matter of time, or did something go sideways?

https://releases.hashicorp.com/terraform/v0.11.10/terraform_v0.11.10_linux_amd64.zip

[lorengordon]

Oof, the url changed, the v is no longer present in the version string, for _any_ version. Was working with the v earlier today, but no longer. Now this is the url:

https://releases.hashicorp.com/terraform/0.11.10/terraform_0.11.10_linux_amd64.zip

[svanharmelen]

Hi @lorengordon, we did not change the download URLs so not sure how you would have been able to download them earlier today using first link you posted.

But good that you found the correct link and can now download the latest release!

[lorengordon]

Well, it had been working for months. Didn't realize it was wrong in any way. Frustrating. Lots of ci configs to update now. :(

[ophintor]

@svanharmelen,

I second @lorengordon. It looks like used to be a 'v'.

The proof is here: https://github.com/hashicorp/terraform/issues/9803#issuecomment-257903082. When I run @sethvargo's command this is what I get:

$ echo "https://releases.hashicorp.com/terraform/$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')/terraform_$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')_darwin_amd64.zip"
https://releases.hashicorp.com/terraform/v0.11.10/terraform_v0.11.10_darwin_amd64.zip

Indeed, the below script used to work for me for ages to update TF to the latest version. It doesn't any more because it picks a 'v':

#!/bin/bash

LATEST_TF_VERSION=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')
TF=${HOME}/bin/terraform
if [ -f ${TF} ] && [ "$(${TF} --version | cut -d'v' -f2)" == "$LATEST_TF_VERSION" ]; then
    echo "Terraform installed and in the latest version"
else
    TF_URL="https://releases.hashicorp.com/terraform/${LATEST_TF_VERSION}/terraform_${LATEST_TF_VERSION}_linux_amd64.zip"
    TF_ZIP="/tmp/tf_${LATEST_TF_VERSION}.zip"
    wget ${TF_URL} -O ${TF_ZIP} && unzip -o ${TF_ZIP} -d ~/bin/
fi

[ophintor]

Another interesting thing is that after updating, Terraform still says there is a new version. I wonder if this is related to the above?

$ terraform --version
Terraform v0.11.10

Your version of Terraform is out of date! The latest version
is v0.11.10. You can update by downloading from www.terraform.io/downloads.html

[svanharmelen]

Thanks for the update @ophintor!

That last issue is a separate issue that we also noticed and are working on. It's caused by a small bug in the version parsing logic of the service that checks your version against the latest version. So it isn't related, but thanks for mentioning!

As for the download URLs, I'm still looking into that but with HashiConf going on things are a bit slow this week.

[paultyng]

@ophintor I think both issues you are mentioning are related, there is a bug in the checkpoint system, which both that script and Terraform use to check if it is out of date. The current version shouldn't have a v in it as far as I understand it, and your second script there, the cut -d'v' -f2 strips the v off, so if it did have a v then it would never get equality with the LATEST_TF_VERSION.

Maybe that is what you are saying in your comment though? But I'm not sure how it would have worked in the past if the checkpoint system returned a v.

[paultyng]

@lorengordon where are you getting that value to sub in to the URLs? From the Checkpoint system also?

[lorengordon]

@paultyng Yep, exactly. I get the current version from checkpoint and sub the value returned into a templated url to get the URL to download the file.

[svanharmelen]

@ophintor @lorengordon so you are not talking about the download URLs themselves, but about the current_version value returned by checkpoint which you use to construct the download URL. Now it all makes sense :wink:

I understand that these issues may seem related, but they really aren't. The first issue was a version parsing issue which was solved by updating the code yesterday. The second issue (v being part of the current_version value) turned out to be a (very unfortunate) typo 😞

That typo is now also corrected, so everything should be fixed and back to normal now. Again sorry for any inconvenience caused!

[lorengordon]

Yep, makes sense now, thanks!

[elops]

I'm seeing this problem still in latest 'terraform' for x86_64

$ terraform -v
Terraform v0.11.10
+ provider.aws v1.50.0

Should be fairly easy to reproduce, just clone
https://github.com/wardviaene/terraform-course.git

minimal setup in demo-1 directory; once I setup my variables in 'terraform.tfvars' when trying to plan I get

$ terraform plan

Error: aws_instance.example: 1 error(s) occurred:

* aws_instance.example: lookup: lookup failed to find '' in:

${lookup(var.AMIS, var.AWS_REGION)}

renaming 'terraform.tfvars' to '.auto.tfvars' does not exit with error, rather prompts me to supply the values for defined variables.

$ terraform plan
var.AWS_ACCESS_KEY
  Enter a value:

[svanharmelen]

@elops I cannot reproduce this problem. Could you maybe share (without the actual keys in there of course) your terraform.tfvars file content?

➜  demo-1 git:(master) ✗ ls -la
total 32
drwxr-xr-x   7 sander  staff   224 Dec  6 09:58 .
drwxr-xr-x  34 sander  staff  1088 Dec  6 09:58 ..
drwxr-xr-x   3 sander  staff    96 Dec  6 09:54 .terraform
-rw-r--r--   1 sander  staff   123 Dec  6 09:50 instance.tf
-rw-r--r--   1 sander  staff   135 Dec  6 09:50 provider.tf
-rw-r--r--   1 sander  staff    46 Dec  6 09:56 terraform.tfvars
-rw-r--r--   1 sander  staff   254 Dec  6 09:50 vars.tf

➜  demo-1 git:(master) ✗ cat terraform.tfvars
AWS_ACCESS_KEY = "xxx"
AWS_SECRET_KEY = "xxx"

➜  demo-1 git:(master) ✗ ../terraform -v
Terraform v0.11.10
+ provider.aws v1.51.0

➜  demo-1 git:(master) ✗ ../terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

------------------------------------------------------------------------

Error: Error running plan: 1 error(s) occurred:

* provider.aws: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
        status code: 403, request id: 1eb56f8a-f935-11e8-a0a3-67f68d5747b1

Of course the token error is because of my dummy token, but this shows it works as expected. And it still works when I rename the file terraform.tfvars to my_custom_var_file.auto.tfvars, so maybe its related to the actual file content?

[elops]

@svanharmelen indeed, I had provider set to empty string in my terraform.tfvars. Removed it and it works as expected.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.