Terraform-provider-aws: Error adding dead_letter_config to lambda function

[veetow]

Terraform Version

0.10.2

Affected Resource(s)

• aws_sqs_queue
• aws_iam_role_policy
• aws_lambda_function

Overview and Expected Behavior

I am attempting to add a dead_letter_config to an existing Lambda function. In doing so, I'm adding the aws_iam_role_policy and aws_sqs_queue resources at the same time.

The plan looks correct, but upon applying the plan, we get the error below. A second plan/apply will succeed. As such, I believe the issue I'm hitting has something to do with the interaction and/or dependency of these resources.

Diff of Terraform Code Changes

diff --git a/terraform/main.tf b/terraform/main.tf
index 0230216..1fade7d 100644
--- a/terraform/main.tf
+++ b/terraform/main.tf
@@ -7,8 +7,6 @@ terraform {
   }
 }

-# TODO: add dead_letter_config
-
 data "aws_subnet_ids" "private_app" {
   vpc_id = "${var.vpc_id}"

@@ -87,6 +85,13 @@ resource "aws_iam_role_policy" "lambda_ddns_delete" {
       "Resource": [
         "arn:aws:s3:::${var.lambda_s3_bucket}/${var.lambda_s3_key}"
       ]
+    },
+    {
+      "Action": [
+        "sqs:SendMessage"
+      ],
+      "Effect": "Allow",
+      "Resource": "${aws_sqs_queue.dlq.arn}"
     }
   ]
 }
@@ -121,6 +126,10 @@ resource "aws_s3_bucket_object" "lambda_ddns_delete" {
   acl    = "bucket-owner-full-control"
 }

+resource "aws_sqs_queue" "dlq" {
+  name = "${var.name}-dlq"
+}
+
 resource "aws_lambda_function" "lambda_ddns_delete" {
   function_name    = "${var.name}"
   s3_bucket        = "${aws_s3_bucket_object.lambda_ddns_delete.bucket}"
@@ -137,6 +146,10 @@ resource "aws_lambda_function" "lambda_ddns_delete" {
     security_group_ids = ["${aws_security_group.lambda_ddns_delete.id}"]
   }

+  dead_letter_config {
+    target_arn = "${aws_sqs_queue.dlq.arn}"
+  }
+
   tags {
     Name       = "${var.name}"
     managed_by = "Terraform"
diff --git a/terraform/outputs.tf b/terraform/outputs.tf
index cfbe193..4e3dba2 100644
--- a/terraform/outputs.tf
+++ b/terraform/outputs.tf
@@ -13,3 +13,7 @@ output "cloudwatch_event_target_id" {
 output "lambda_function_id" {
   value = "${aws_lambda_function.lambda_ddns_delete.id}"
 }
+
+output "sqs_dlq_arn" {
+  value = "${aws_sqs_queue.dlq.arn}"
+}

Plan output

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_s3_bucket_object.lambda_ddns_delete: Refreshing state... (ID: dns/lambda_ddns_delete.zip)
aws_sqs_queue.dlq: Refreshing state... (ID: https://sqs.us-east-1.amazonaws.com/256428178804/lambda_ddns_delete-dlq)
aws_cloudwatch_event_rule.ec2_instance_terminating: Refreshing state... (ID: ec2_instance_terminating)
aws_security_group.lambda_ddns_delete: Refreshing state... (ID: sg-3e936c4e)
aws_iam_role.lambda_ddns_delete: Refreshing state... (ID: lambda_ddns_delete)
data.aws_subnet_ids.private_app: Refreshing state...
aws_iam_role_policy_attachment.AmazonEC2ReadOnlyAccess: Refreshing state... (ID: lambda_ddns_delete-0051f1fed988efeb279ba2e404)
aws_iam_role_policy_attachment.AWSLambdaVPCAccessExecutionRole: Refreshing state... (ID: lambda_ddns_delete-006acdf6949441480ca12baddb)
aws_iam_role_policy_attachment.AmazonVPCReadOnlyAccess: Refreshing state... (ID: lambda_ddns_delete-0051f1fed988efeb279ba2e402)
aws_iam_role_policy_attachment.CloudWatchLogsFullAccess: Refreshing state... (ID: lambda_ddns_delete-0051f1fed988efeb279ba2e403)
aws_security_group_rule.allow_all_egress: Refreshing state... (ID: sgrule-3074313034)
aws_security_group_rule.allow_all_ingress: Refreshing state... (ID: sgrule-3755793044)
random_shuffle.app_subnets: Refreshing state... (ID: -)
aws_iam_role_policy.lambda_ddns_delete: Refreshing state... (ID: lambda_ddns_delete:lambda_ddns_delete-policy)
aws_lambda_function.lambda_ddns_delete: Refreshing state... (ID: lambda_ddns_delete)
aws_lambda_permission.allow_cloudwatch: Refreshing state... (ID: AllowExecutionFromCloudWatch)
aws_cloudwatch_event_target.ec2_instance_termination: Refreshing state... (ID: ec2_instance_terminating-terraform-006acdf6949441480ca12baddc)
The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: sandbox.terraform.plan

  + aws_sqs_queue.dlq
      arn:                               "<computed>"
      content_based_deduplication:       "false"
      delay_seconds:                     "0"
      fifo_queue:                        "false"
      kms_data_key_reuse_period_seconds: "<computed>"
      max_message_size:                  "262144"
      message_retention_seconds:         "345600"
      name:                              "lambda_ddns_delete-dlq"
      policy:                            "<computed>"
      receive_wait_time_seconds:         "0"
      visibility_timeout_seconds:        "30"

  ~ aws_iam_role_policy.lambda_ddns_delete
      policy: "{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:*\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::webmd-infrastructure-repo/dns/lambda_ddns_delete.zip\"\n            ]\n        }\n    ]\n}" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:*\"\n      ],\n      \"Resource\": [\n        \"arn:aws:s3:::${var.lambda_s3_bucket}/${var.lambda_s3_key}\"\n      ]\n    },\n    {\n      \"Action\": [\n        \"sqs:SendMessage\"\n      ],\n      \"Effect\": \"Allow\",\n      \"Resource\": \"${aws_sqs_queue.dlq.arn}\"\n    }\n  ]\n}\n"

  ~ aws_lambda_function.lambda_ddns_delete
      dead_letter_config.#:            "0" => "1"
      dead_letter_config.0.target_arn: "" => "${aws_sqs_queue.dlq.arn}"


Plan: 1 to add, 2 to change, 0 to destroy.

Actual Behavior

aws_sqs_queue.dlq: Creating...
  arn:                               "" => "<computed>"
  content_based_deduplication:       "" => "false"
  delay_seconds:                     "" => "0"
  fifo_queue:                        "" => "false"
  kms_data_key_reuse_period_seconds: "" => "<computed>"
  max_message_size:                  "" => "262144"
  message_retention_seconds:         "" => "345600"
  name:                              "" => "lambda_ddns_delete-dlq"
  policy:                            "" => "<computed>"
  receive_wait_time_seconds:         "" => "0"
  visibility_timeout_seconds:        "" => "30"
aws_sqs_queue.dlq: Creation complete (ID: https://sqs.us-east-1.amazonaws.com/256428178804/lambda_ddns_delete-dlq)
aws_iam_role_policy.lambda_ddns_delete: Modifying... (ID: lambda_ddns_delete:lambda_ddns_delete-policy)
  policy: "{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:*\"\n        
    ],\n            \"Resource\": [\n                \"arn:aws:s3:::webmd-infrastructure-repo/dns/lambda_ddns_delete.zip\"\n            ]\n        }\n    ]\n}" => "{\n  \"Version
\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:*\"\n      ],\n      \"Resource\": [\n        \"arn:aws:s3:::webmd
-infrastructure-repo/dns/lambda_ddns_delete.zip\"\n      ]\n    },\n    {\n      \"Action\": [\n        \"sqs:SendMessage\"\n      ],\n      \"Effect\": \"Allow\",\n      \"Resou
rce\": \"arn:aws:sqs:us-east-1:256428178804:lambda_ddns_delete-dlq\"\n    }\n  ]\n}\n"
aws_lambda_function.lambda_ddns_delete: Modifying... (ID: lambda_ddns_delete)
  dead_letter_config.#:            "0" => "1"
  dead_letter_config.0.target_arn: "" => "arn:aws:sqs:us-east-1:256428178804:lambda_ddns_delete-dlq"
aws_iam_role_policy.lambda_ddns_delete: Modifications complete (ID: lambda_ddns_delete:lambda_ddns_delete-policy)
Error applying plan:

1 error(s) occurred:

* aws_lambda_function.lambda_ddns_delete: 1 error(s) occurred:

* aws_lambda_function.lambda_ddns_delete: Error modifying Lambda Function Configuration lambda_ddns_delete: InvalidParameterValueException: The provided execution role does not have permissions to call SendMessage on SQS
        status code: 400, request id: d4fd9d46-89a1-11e7-9d23-5d583bc8e1b3

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[gazoakley]

I've seen this before with aws_lambda_function - I think it's a case of IAM roles taking time to propagate around AWS - from http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#launch-instance-with-role:

After you create an IAM role, it may take several seconds for the permissions to propagate.

A possible but nasty workaround is to use a provisioner on the IAM role resource to sleep 10 seconds:

resource "aws_iam_role_policy_attachment" "lambda_sns_attach" {
  role       = "${aws_iam_role.lambda_execute_role.name}"
  policy_arn = "${aws_iam_policy.sns_policy.arn}"

  # Ensures IAM policy is created and available before creating lambda functions
  provisioner "local-exec" {
    command = "sleep 10"
  }
}

Creating/modifying a lambda function should probably retry a couple of times if it spots an error regarding IAMs roles. There are other places in Terraform that do this already.

[veetow]

That _is_ a nasty workaround. 🥇

Is this something that is likely to be fixed in the provider itself at some point?

[Ninir]

HI folks,

the lambda function already does that on the creation part but isn't for the update.

For the creation part, it would be to add another if condition with the specified error.
For the update part, it would be about adding the retry function, etc.

Is anyone willing to contribute that? we would be happy to review! :)

[gazoakley]

@Ninir - should be an easy fix, might be able to have a look at it on Sunday :-)

[bflad]

@gazoakley are you still able to take a look at/implement the retry functionality for this? Otherwise, I can easily get this in. We have recently added the update Lambda function retry error handling for EC2 throttling in #2964 and this should be a very similar code update.

[gazoakley]

@bflad sorry! I'm not able to do much this week so feel free to grab it 😄

[bflad]

PR submitted: #3116

[bflad]

Retries for IAM eventual consistency in Lambda function updates have been added in master and will be released in v1.8.0 of the provider -- hopefully tomorrow! 🎉

[bflad]

This has been released in terraform-provider-aws version 1.8.0. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!