Aws-sdk-go: IAM ListRoles SerializationError: failed decoding Query response

[bflad]

Version of AWS SDK for Go?

1.19.6

Version of Go (go version)?

1.11.4

What issue did you see?

It appears some IAM API calls are having intermittent issues today. Seems like I'm not the only one according to #1628 (closed, so opening new top level issue). The below error seems to be caused by an empty 200 response.

SDK error message:

DEBUG: Unmarshal Response iam/ListRoles failed, not retrying, error SerializationError: failed decoding Query response

Full debug logging:

2019/04/05 01:58:56 [DEBUG] [aws-sdk-go] DEBUG: Request iam/ListRoles Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: iam.amazonaws.com
User-Agent: aws-sdk-go/1.19.6 (go1.11.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.14-dev
Content-Length: 35
Authorization: AWS4-HMAC-SHA256 Credential=*******/20190405/us-east-1/iam/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=526ac0bb4e208337c7fc098bfae636addc6f1b407929c6812e0a97c779653c66
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20190405T015856Z
Accept-Encoding: gzip

Action=ListRoles&Version=2010-05-08
-----------------------------------------------------
2019/04/05 01:58:57 [DEBUG] [aws-sdk-go] DEBUG: Response iam/ListRoles Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Fri, 05 Apr 2019 01:58:56 GMT
Vary: accept-encoding
X-Amzn-Requestid: 5f5a477f-5746-11e9-b69c-23c001493449


-----------------------------------------------------
2019/04/05 01:58:57 [DEBUG] [aws-sdk-go] <ListRolesResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <ListRolesResult>
    <IsTruncated>true</IsTruncated>
    <Marker>AAFXRH/Fg3d3L57FTy98/0ERvnVQNasHUOFNw696nsvMnUIA8Z9cuUa+A797Oy6QbmF6okGE4DK5tfOcJmO7NJa3D0MFQbGLrzvxEzmX+1tQOw==</Marker>
    <Roles>
-- 100 <member>...</member> elements OMITTED --
    </Roles>
  </ListRolesResult>
  <ResponseMetadata>
    <RequestId>5f5a477f-5746-11e9-b69c-23c001493449</RequestId>
  </ResponseMetadata>
</ListRolesResponse>
2019/04/05 01:58:57 [DEBUG] [aws-sdk-go] DEBUG: Request iam/ListRoles Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: iam.amazonaws.com
User-Agent: aws-sdk-go/1.19.6 (go1.11.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.14-dev
Content-Length: 167
Authorization: AWS4-HMAC-SHA256 Credential=*******/20190405/us-east-1/iam/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=acf4f83351833219583b97bd523a4f31d680a75db250e9367c7b4d779d3d4b50
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20190405T015857Z
Accept-Encoding: gzip

Action=ListRoles&Marker=AAFXRH%2FFg3d3L57FTy98%2F0ERvnVQNasHUOFNw696nsvMnUIA8Z9cuUa%2BA797Oy6QbmF6okGE4DK5tfOcJmO7NJa3D0MFQbGLrzvxEzmX%2B1tQOw%3D%3D&Version=2010-05-08
-----------------------------------------------------
2019/04/05 01:58:57 [DEBUG] [aws-sdk-go] DEBUG: Response iam/ListRoles Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Fri, 05 Apr 2019 01:58:57 GMT
Vary: accept-encoding
X-Amzn-Requestid: 4401ad10-dcae-4705-b0bf-bfa4d24c9842
X-Amzn-Requestid: 4401ad10-dcae-4705-b0bf-bfa4d24c9842


-----------------------------------------------------
2019/04/05 01:58:57 [DEBUG] [aws-sdk-go] 
2019/04/05 01:58:57 [DEBUG] [aws-sdk-go] DEBUG: Unmarshal Response iam/ListRoles failed, not retrying, error SerializationError: failed decoding Query response
    status code: 200, request id: 4401ad10-dcae-4705-b0bf-bfa4d24c9842
caused by: unexpected EOF

Steps to reproduce

Simplified code:

    roles := make([]*iam.Role, 0)

    err = conn.ListRolesPages(&iam.ListRolesInput{}, func(page *iam.ListRolesOutput, lastPage bool) bool {
        roles = append(roles, page.Roles...)

        return !lastPage
    })

    if err != nil {
        return fmt.Errorf("Error retrieving IAM Roles: %s", err)
    }

Is there anything we can do to handle this error better via configuring AWS Go SDK handlers on the client side or potentially within the SDK itself? Would automatically retrying certain "read only" API calls on SerializationError be advisable?

[tannirurahul]

we are experiencing the same issue in us-east-1. here is the piece of log that can help

DEBUG: Unmarshal Response iam/GetPolicyVersion failed, not retrying, error SerializationError: failed decoding Query response
2019-04-04T21:30:11.309-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: status code: 200, request id: 80a3a858-81e1-4730-99da-cc4224a88690
EBUG] plugin.terraform-provider-aws_v2.4.0_x4: caused by: unexpected EOF

[scsarver]

This is happening in us-eat-1 for IAM for me as well:

2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/04 21:49:57 [DEBUG] [aws-sdk-go] DEBUG: Response iam/GetRolePolicy Details:
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: ---[ RESPONSE ]--------------------------------------
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: HTTP/1.1 200 OK
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Connection: close
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Transfer-Encoding: chunked
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Content-Type: text/xml
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Date: Fri, 05 Apr 2019 02:49:56 GMT
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Vary: accept-encoding
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: X-Amzn-Requestid: XXXXXXXXXXXXXXXXXXXX
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: X-Amzn-Requestid: XXXXXXXXXXXXXXXXXXXX
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: -----------------------------------------------------
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/04 21:49:57 [DEBUG] [aws-sdk-go]
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/04 21:49:57 [DEBUG] [aws-sdk-go] DEBUG: Unmarshal Response iam/GetRolePolicy failed, not retrying, error SerializationError: failed decoding Query response
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4:   status code: 200, request id: XXXXXXXXXXXXXXXXXXXX
2019-04-04T21:49:57.016-0500 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: caused by: unexpected EOF
2019/04/04 21:49:57 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_iam_role_policy.XXXXXXXXXXXXXXXXXXXX: Error reading IAM policy XXXXXXXXXXXXXXXXXXXX from role XXXXXXXXXXXXXXXXXXXX: SerializationError: failed decoding Query response
    status code: 200, request id: XXXXXXXXXXXXXXXXXXXX
caused by: unexpected EOF
2019/04/04 21:49:57 [ERROR] root: eval: *terraform.EvalSequence, err: aws_iam_role_policy.XXXXXXXXXXXXXXXXXXXX: Error reading IAM policy XXXXXXXXXXXXXXXXXXXX from role XXXXXXXXXXXXXXXXXXXX: SerializationError: failed decoding Query response
    status code: 200, request id: XXXXXXXXXXXXXXXXXXXX
caused by: unexpected EOF
2019/04/04 21:49:57 [TRACE] [walkRefresh] Exiting eval tree: aws_iam_role_policy.XXXXXXXXXXXXXXXXXXXX

[devonbleak]

Also happening intermittently on iam.GetPolicyVersion and iam.GetSamlProvider among other calls here. Looks like the service is just returning empty responses.

Example debug output:

2019-04-04T20:03:35.697-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: -----------------------------------------------------
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/04 20:03:35 [DEBUG] [aws-sdk-go] DEBUG: Response iam/GetPolicyVersion Details:
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: ---[ RESPONSE ]--------------------------------------
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: HTTP/1.1 200 OK
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Connection: close
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Transfer-Encoding: chunked
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Content-Type: text/xml
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Date: Fri, 05 Apr 2019 03:03:34 GMT
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: Vary: accept-encoding
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: X-Amzn-Requestid: 4a0735dc-61eb-41ba-a1f4-791211a7ea67
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: X-Amzn-Requestid: 4a0735dc-61eb-41ba-a1f4-791211a7ea67
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: -----------------------------------------------------
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/04 20:03:35 [DEBUG] [aws-sdk-go]
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: 2019/04/04 20:03:35 [DEBUG] [aws-sdk-go] DEBUG: Unmarshal Response iam/GetPolicyVersion failed, not retrying, error SerializationError: failed decoding Query response
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4:   status code: 200, request id: 4a0735dc-61eb-41ba-a1f4-791211a7ea67
2019-04-04T20:03:35.735-0700 [DEBUG] plugin.terraform-provider-aws_v2.4.0_x4: caused by: unexpected EOF

You can see the line below the response headers where the response body should be contains no data. Have tried on multiple independent networks and get similar results, always from iam service.

[jasdel]

Thanks for the reporting this issue. We're investigating this issue. IAM is aware of this issue and is investigating the cause. Please let us know if you are still experiencing this issue.

[devonbleak]

So far I'm unable to reproduce any more.

[jasdel]

Thanks for the update. Please let us know if you encounter this issue again. If you do, it would be helpful to include the requestIDs of the failed request, along with region, environment, and method of retrieving credentials your application is using.

[diehlaws]

@bflad @tannirurahul @scsarver Are you still running into issues listing IAM roles using the AWS SDK for Go?

[scsarver]

@diehlaws I have not experienced this since It was reported 3 weeks ago.

[diehlaws]

Thanks for confirming @scsarver; this seems to have been a transient issue with the service rather than a problem with the AWS SDK for Go. As @jasdel mentioned above, if anyone encounters this issue again please leave a comment including Request IDs, affected region(s), and environment related to the failed requests so that we can re-open the issue and further investigate this behavior.