Terraform-provider-aws: aws_iam_role_policy SerializationError

[texascloud]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.1
aws-provider 1.59

Affected Resource(s)

• aws_iam_role_policy

Terraform Configuration Files

resource "aws_iam_role" "node_iam_role" {
  name = "node_iam_role-paul"
  force_detach_policies = true
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "node_iam_role_policy" {
  name = "node_iam_role_policy-blart"
  role = "${aws_iam_role.node_iam_role.id}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::kubernetes-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["route53:*"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:BatchGetImage"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["elasticloadbalancing:*"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": ["ec2:*"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": ["autoscaling:*"],
      "Resource": ["*"]
    },
    {
       "Effect": "Allow",
       "Action": ["ses:SendEmail", "ses:SendRawEmail"],
       "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["rds:*"],
      "Resource": "*"
    }
  ]
}
EOF
}

Debug Output

https://gist.github.com/CamelCaseNotation/50a68d67721a16b45059b893aa669669

Expected Behavior

This Terraform configuration is used to perform applies many times a day and typically succeeds. I assume this is some sort of intermittent issue? Or perhaps having to do with the aws-go-sdk library?
iam role results returned correctly to Terraform

Actual Behavior

Terraform apply failed

Steps to Reproduce

1. terraform apply

[texascloud]

I see this comment in another open issue: https://github.com/terraform-providers/terraform-provider-aws/issues/7075#issuecomment-452850177 Maybe same deal?

[devonbleak]

I'm getting SerializationError with unexpected EOF on multiple resources including:

• aws_iam_saml_provider
• aws_iam_user_policy
• aws_iam_policy
• aws_iam_role_policy

Terraform v0.11.11
+ provider.archive v1.0.0
+ provider.aws v2.4.0

Also tried downgrading aws provider to v2.3.0 and getting the same issues.

The actual resources throwing errors seems to change on every run.

2 things I think this could be:
1. Our Palo Altos are throttling the outbound connections to AWS (I'm suddenly running into throttling issues trying to terraform init -upgrade in large workspaces again) - if you're also behind a PAN firewall then a signature update could be breaking things. Not the Palos - tried from a separate network.

1. Issue with AWS APIs

Given that these resources were definitely fine on these provider versions yesterday it doesn't seem like this is necessarily an issue with TF or the AWS provider.

[zentavr]

I have the same issue all day long today.

Terarform 0.11
AWS Provider: 2.4.0

[zentavr]

Something from my Jenkins:

04:54:54  2019-04-05T01:54:54.789Z [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/05 01:54:54 [DEBUG] [aws-sdk-go] {"Policy":"{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"Allow-stage-device_info-invoke\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-west-2:012432584802:function:stage_device_info\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-west-2:012432584802:rq3tubf6bg/*/*/*\"}}}]}","RevisionId":"28ea612b-28b9-46ec-8f3f-b5e3be2604d6"}
04:54:54  2019-04-05T01:54:54.789Z [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/05 01:54:54 [DEBUG] Received 1 statements in Lambda policy: [{map[ArnLike:map[AWS:SourceArn:arn:aws:execute-api:us-west-2:012432584802:rq3tubf6bg/*/*/*]] lambda:InvokeFunction arn:aws:lambda:us-west-2:012432584802:function:stage_device_info Allow map[Service:apigateway.amazonaws.com] Allow-stage-device_info-invoke}]
04:54:54  2019-04-05T01:54:54.790Z [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/05 01:54:54 [ERR] Error getting Lambda Qualifier: Invalid ARN or otherwise unable to get qualifier from ARN ("arn:aws:lambda:us-west-2:012432584802:function:stage_device_info")

[jpterry]

This also started happening out of the blue for us.
We use a very static process, nothing has changed recently. First noticed today at 5:59 PDT.
Terraform 0.11.11 AWS 1.60.0 (also tried 1.40.0).

[bflad]

At least for me, this was caused by a completely empty response body from IAM.

I have opened an upstream AWS Go SDK issue here: https://github.com/aws/aws-sdk-go/issues/2549

I would advise opening your own AWS Support cases and cross linking this issue and the above one as that should help prioritize the response from the service team.

[namliz]

I just tried with TF_LOG=debug and it happened to have worked after sporadically getting this error all day.

@bflad I'm a little confused if this is a terraform bug, an aws-sdk-go bug or an aws api breakage?

[zentavr]

@zilman I think it was an accident that it worked. I was able to deploy my environment 1 time with dozens of trials.

[devonbleak]

It appears to be an API bug. It's been consistently working for me for the past 10 minutes since AWS asked for confirmation it was still broken in the other issue.

[zentavr]

ok, few seconds ago was able to deploy the AWS env. no errors. the code is the same as it was all day long.

[severity1]

I think any API request actually works its just that the response body is not what terraform or the provider is expecting.

[eskp]

Confirm working here now also.