Aws-sdk-go: AWS Go SDK v1.28.0 Code Generated Error Types Missing Error Message

Created on 20 Jan 2020 · 1Comment · Source: aws/aws-sdk-go

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

Version 1.28.5

Version of Go (`go version`)?

Go 1.13

What issue did you see?

Changes in v1.28.0 of the AWS Go SDK (imported and released by Terraform AWS Provider version 2.45.0) introduced code generated types for certain service error shapes, which can be used to extract additional information provided by the API response. Prior to these code generated types, most (if not all) of these errors returned the correct error messaging via Message() via the message protocol response field. However after this change, since some of these error shapes in the service API models are incorrectly missing the Message structure member, this causes the Message_ (Go struct type) to be missing, which causes the error message via Message() to always be empty.

An example of this missing error messaging can be seen with codebuild service InvalidInputException shape:

https://github.com/aws/aws-sdk-go/blob/6627e8938cdef5476dec0510b524258d3bf3983e/models/apis/codebuild/2016-10-06/api-2.json#L998-L1003

https://github.com/aws/aws-sdk-go/blob/6627e8938cdef5476dec0510b524258d3bf3983e/service/codebuild/api.go#L5333-L5336

Debug output from the application shows the error message being returned, but it does not seem possible to access given the above (note missing error message after error code and colon on last line):

2020/01/20 12:59:15 [DEBUG] [aws-sdk-go] DEBUG: Response codebuild/CreateProject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 167
Content-Type: application/x-amz-json-1.1
Date: Mon, 20 Jan 2020 17:59:14 GMT
X-Amzn-Requestid: 2f589026-d531-458d-b851-86bd4afb5cd0


-----------------------------------------------------
2020/01/20 12:59:15 [DEBUG] [aws-sdk-go] {"__type":"InvalidInputException","message":"CodeBuild is not authorized to perform: sts:AssumeRole on arn:aws:iam::--OMITTED--:role/tf-acc-test-4387558838057379294"}
2020/01/20 12:59:15 [DEBUG] [aws-sdk-go] DEBUG: Validate Response codebuild/CreateProject failed, attempt 0/25, error InvalidInputException:

Full SDK analysis of this new behavior was performed here: https://github.com/terraform-providers/terraform-provider-aws/issues/11642#issuecomment-575820896

In terms of the AWS Go SDK, this change potentially affected the following services:

apigatewaymanagementapi
cloud9
cloudhsm
cloudtrail
cloudwatchevents
cloudwatchlogs
codebuild
codecommit
codedeploy
codepipeline
codestar
configservice
dax
directconnect
ecs
elasticsearchservice
elastictranscoder
emr
eventbridge
lexmodelbuildingservice
managedblockchain
pinpointemail
rdsdataservice
rekognition
servicecatalog
sesv2
sms
ssm
ssooidc
textract
transfer
waf
wafregional
workdocs

In terms of the Terraform AWS Provider, this change broke the following behaviors which were dependent on checking the previously available error messages:

In aws_cloud9_environment_ec2 resource, unable to retry for IAM eventual consistency on NotFoundException during creation (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_cloud9_environment_ec2.go#L96-L98)
In aws_cloudtrail resource, unable to retry for CloudWatch Logs eventual consistency on InvalidCloudWatchLogsRoleArnException and InvalidCloudWatchLogsLogGroupArnException during creation or update (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_cloudtrail.go#L186-L191)
In aws_cloudwatch_log_destination resource, unable to retry for InvalidParameterException with test message delivery during creation or update (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_cloudwatch_log_destination.go#L70-L72)
In aws_cloudwatch_log_subscription resource, unable to retry for InvalidParameterException with test message delivery and Lambda Function execution during creation (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_cloudwatch_log_subscription_filter.go#L71-L76)
In aws_codebuild_project resource, unable to retry for IAM eventual consistency on InvalidInputException during creation and update (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_codebuild_project.go#L665-L669)
In aws_codedeploy_deployment_group resource, unable to retry for SNS Topic eventual consistency on InvalidTriggerConfigException during creation and update (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_codedeploy_deployment_group.go#L783-L788)
In aws_config_delivery_channel resource, unable to retry for running Config Configuration Recorder on LastDeliveryChannelDeleteFailedException during deletion (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_config_delivery_channel.go#L172-L174)
In aws_ecs_service resource, unable to retry for cross-service eventual consistency on InvalidParameterException during creation and update (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_ecs_service.go#L524-L533)
In aws_elasticsearch_domain resource, unable to retry for various eventual consistency errors on InvalidTypeException / ValidationException during creation (https://github.com/terraform-providers/terraform-provider-aws/blob/96840caea28e3abf72d0d994550d2ee7d4b7fc45/aws/resource_aws_elasticsearch_domain.go#L459-L474)

Historically, we have preferred to sometimes check specific error messaging from the service API responses since the error code itself may not be descriptive enough to determine if the error could/should be retried. While the service APIs should likely introduce more granular error codes and fix their API models, the unexpected removal of the error messaging represents a fairly problematic change in the AWS Go SDK since it was previously supported by many services.

Is it possible for these errors to try and return a message, if available, even if the Message member is undefined in the API model?

References:

https://github.com/aws/aws-sdk-go/pull/3062
https://github.com/terraform-providers/terraform-provider-aws/issues/11682
https://github.com/terraform-providers/terraform-provider-aws/issues/11642
https://github.com/terraform-providers/terraform-provider-aws/issues/7725

Steps to reproduce

From https://github.com/terraform-providers/terraform-provider-aws:

$ TF_ACC=1 go test ./aws -v -count 1 -timeout 120m -parallel 20 -run='TestAccAWSCodeBuildProject_basic'

bug

Source

bflad

Most helpful comment

Thanks for reporting this bug, and the in depth description of the problem. It looks like the SDK's generation needs to ensure all errors have a Message_ member regardless if the error is modeled as having the value.

We'll work on getting a fix out for this bug.