Terraform: Consistent failure when deploying a generic module to all AWS regions

[jonathan-kosgei]

I have a main.tf that looks like

module "us-east-1" {
    source = "./modules/multi-region"
    region = "us-east-1"
    access_key = "${var.access_key}"
    secret_key = "${var.secret_key}"
    role       = "${aws_iam_role.lambda-role.arn}"
}
module "us-east-2" {
    source = "./modules/multi-region"
    region = "us-east-2"
    access_key = "${var.access_key}"
    secret_key = "${var.secret_key}"
    role       = "${aws_iam_role.lambda-role.arn}"
}

# The above blocks are repeated 11 more times for each AWS region

The main part of the module contains

provider "aws" {
    region = "${var.region}"
    access_key = "${var.access_key}"
    secret_key = "${var.secret_key}"
}
resource "aws_lambda_function" "helloworld" {
  s3_bucket        = "helloworld-zip"
  s3_key           = "${aws_s3_bucket_object.helloworld.id}"
  function_name    = "helloworld"
  role             = "${var.role}"
  handler          = "helloworld.helloworld"
  runtime          = "python2.7"
  publish          = "true"
}

From the above you can see I'm trying to create a lambda function in each region by passing in a different region each time to the provider in the module.

However, after running this dozens of times, I've found that terraform will consistently fail to create the resource in 3 regions. Always 3 random regions, could be us-east-1, us-east-2 then sa-east-1 or eu-west-1, eu-west-2 then ca-central-1a.

I have seen this repeatedly and consistently with Terraform v0.10.4.

The thing is though, runningterraform plan on a fresh try (before running terraform apply) I'll see all 13 regions listed.

After the first run 10 regions will have the new function while 3 will have failed with a Name conflict claiming the function already exists in that region.

I'd maually check via the CLI (after configuring my region) and on the dashboard - in that region - and find that the function DOES NOT exist.

Running terraform plan at this stage will correctly show that 3 more functions need to be created in 3 more regions, but running terraform apply will consistently fail.

Even if I save the correctly generated plan and run terraform apply on it.

I'll still get the same Name Conflict function already exists error.

[lholman]

@jonathan-kosgei Are you using the same function name each time, maybe you're hitting an eventual consistency issue with the deployment package in S3 as you tear it down and redeploy? Have you tried adding some randomness to the name?

[jonathan-kosgei]

@lholman you mean the helloworld function name? In my earlier tries I was passing in the s3 object to the module, the s3 bucket and upload was done in the main.tf (in the root, outside the module) I wasn't actually creating a bucket per region with the function, though I did eventually try that

[jonathan-kosgei]

@lholman hey! I added the region to the end of the function name and it worked! Thanks!!

Something like

resource "aws_lambda_function" "helloworld" {
  s3_bucket        = "helloworld-zip"
  s3_key           = "${aws_s3_bucket_object.helloworld.id}"
  function_name    = "helloworld-${var.region}"
  role             = "${var.role}"
  handler          = "helloworld.helloworld"
  runtime          = "python2.7"
  publish          = "true"
}

And voila, I'm able to have my function in every region.

[lholman]

@jonathan-kosgei That's the puppy. Check out http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel looks like you were overwriting the same file each time

[apparentlymart]

Hi @jonathan-kosgei!

It sounds like you found a working approach here, so I'm going to close this. I'm not entirely sure I follow what changed here, but if I understood correctly it sounds like the problem was on the AWS end after all, with some resources colliding. If you think there _is_ still a bug here, I would encourage opening an issue in the AWS provider repository, though no need to do that if you're satisfied with the solution you found here.

Thanks also for the hints, @lholman! :grinning:

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.