Terraform: Unnecessary destroy of aws_api_gateway_base_path_mapping + incorrect parsing of error message from AWS response

[crumley]

I have a basic terraform lambda + api gateway project. I added aws_api_gateway_base_path_mapping to the project which succeeds the first time it runs. Subsequent runs result in a diff where XXX is the same on each side but for some reason the rest api id is changing from empty to the current id (which has not changed since the first run):

-/+ module.events.aws_api_gateway_base_path_mapping.dns
    api_id:      "" => "lecfdog152" (forces new resource)
    domain_name: "XXX" => "XXX"

Once this plan is applied there is an error about decoding the error response from AWS:

* aws_api_gateway_base_path_mapping.dns: SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value

The real error response is XML and is visible in the including DEBUG output.

Terraform Version

terraform 0.7.4

Affected Resource(s)

aws_api_gateway_base_path_mapping

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

Debug Output

https://gist.github.com/crumley/e4090d3e9e59efadf4d652e4fc5a4982

Expected Behavior

Since no changes to this resource or a dependent resource have occured I would expect no API call to be required. In the event that one is required it should succeed.

Actual Behavior

Got an error message caused by the underlying AWS 403 response.

* aws_api_gateway_base_path_mapping.dns: SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value

Steps to Reproduce

I suspect the problem would occur even with the standard example from the documentation as mine has not been modified:

https://www.terraform.io/docs/providers/aws/r/api_gateway_base_path_mapping.html

Extra

Note that I can delete the base mapping with the AWS console and the same credentials I am running terraform with.

[crumley]

FYI also following up with AWS support to see if they can give more insight into why the delete base mapping call is failing. I will report back once I have more information. Meanwhile trying to find a workaround that will prevent the resource from being destroyed / recreated so I can continue using the resource.

[BerndWessels]

@crumley hi, I ran into the same problem. Have you found a workaround?

[crumley]

Nothing yet... I ended up commenting out the aws_api_gateway_base_path_mapping and manually deleting the first entry it was able to make so future runs of terraform don't bomb. I haven't been able to get help from AWS on the error but am still trying and will report back.

[apparentlymart]

Over in #8466 @richardc asserted that the REST API Id (among other things) can be nil sometimes. I'm not entirely sure what that would mean, since after all the purpose of the base path mapping is to connect a path and domain name with a REST API, but that would be consistent with the behavior you are seeing here: seems like Terraform is getting back nil when it refreshes the mapping and storing that in state as the empty string.

If your debug output also includes the request and response bodies for the GetBasePathMapping call made during refresh that could confirm whether my theory is true.

[BerndWessels]

@crumley @apparentlymart @stack72 Hi,

Terraform v0.7.5

here is the error I get:

* aws_api_gateway_base_path_mapping.SeoWebsite: SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value

and here are the logs you asked for:

    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: aws-provider (internal) 2016/10/11 08:09:35 [DEBUG] [aws-sdk-go] DEBUG: Request apigateway/GetBasePathMapping Details:
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: ---[ REQUEST POST-SIGN ]-----------------------------
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: GET http://apigateway.us-east-1.amazonaws.com/domainnames/seo.domain.nz/basepathmappings/ HTTP/1.1
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Host: apigateway.us-east-1.amazonaws.com
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: User-Agent: terraform/0.7.5 aws-sdk-go/1.4.11 (go1.7; windows; amd64)
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Accept: application/json
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxx
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: X-Amz-Date: 20161010T190935Z
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Accept-Encoding: gzip
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: 


    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: aws-provider (internal) 2016/10/11 08:09:35 [DEBUG] [aws-sdk-go] DEBUG: Response apigateway/GetBasePathMapping Details:
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: ---[ RESPONSE ]--------------------------------------
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: HTTP/1.1 200 OK
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Content-Length: 73
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Content-Type: application/json
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: Date: Mon, 10 Oct 2016 19:09:35 GMT
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: X-Amzn-Requestid: 15780766-8f1d-11e6-8dde-e150b979e3c3
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: {"item":[{"basePath":"(none)","restApiId":"g6h2x0n5hl","stage":"prod"}]}
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:35 [DEBUG] plugin: terraform.exe: -----------------------------------------------------
    2016/10/11 08:09:35 [DEBUG] root: eval: *terraform.EvalWriteState
    2016/10/11 08:09:35 [DEBUG] root: eval: terraform.EvalNoop
    2016/10/11 08:09:35 [DEBUG] root: eval: terraform.EvalNoop
    2016/10/11 08:09:35 [DEBUG] root: eval: terraform.EvalNoop
    2016/10/11 08:09:35 [TRACE] [walkRefresh] Exiting eval tree: aws_api_gateway_base_path_mapping.SeoWebsite



    aws_api_gateway_base_path_mapping.SeoWebsite (destroy) - *terraform.graphNodeExpandedResourceDestroy
    2016/10/11 08:09:44 [DEBUG] vertex root.aws_api_gateway_base_path_mapping.SeoWebsite (destroy): walking
    2016/10/11 08:09:44 [DEBUG] vertex root.aws_api_gateway_base_path_mapping.SeoWebsite (destroy): evaluating
    2016/10/11 08:09:44 [TRACE] [walkApply] Entering eval tree: aws_api_gateway_base_path_mapping.SeoWebsite (destroy)
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalOpFilter
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalSequence
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalReadDiff
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalFilterDiff
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalIf
    2016/10/11 08:09:44 [DEBUG] root: eval: terraform.EvalNoop
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalInstanceInfo
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalGetProvider
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalReadState
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalRequireState
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalIf
    2016/10/11 08:09:44 [DEBUG] root: eval: *terraform.EvalApply
    2016/10/11 08:09:44 [DEBUG] apply: aws_api_gateway_base_path_mapping.SeoWebsite: executing Apply
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: aws-provider (internal) 2016/10/11 08:09:44 [DEBUG] [aws-sdk-go] DEBUG: Request apigateway/DeleteBasePathMapping Details:
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: ---[ REQUEST POST-SIGN ]-----------------------------
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: DELETE http://apigateway.us-east-1.amazonaws.com/domainnames/seo.domain.nz/basepathmappings/ HTTP/1.1
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: Host: apigateway.us-east-1.amazonaws.com
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: User-Agent: terraform/0.7.5 aws-sdk-go/1.4.11 (go1.7; windows; amd64)
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: Accept: application/json
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: Authorization: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: X-Amz-Date: 20161010T190944Z
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: Accept-Encoding: gzip
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:44 [DEBUG] plugin: terraform.exe: -----------------------------------------------------
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: aws-provider (internal) 2016/10/11 08:09:45 [DEBUG] [aws-sdk-go] DEBUG: Response apigateway/DeleteBasePathMapping Details:
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: ---[ RESPONSE ]--------------------------------------
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: HTTP/1.1 403 Forbidden
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: Content-Length: 130
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: Date: Mon, 10 Oct 2016 19:09:44 GMT
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: X-Amzn-Requestid: 1affbdaf-8f1d-11e6-ad4a-331949c56ff9
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: <AccessDeniedException>
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe:   <Message>Unable to determine service/operation name to be authorized</Message>
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: </AccessDeniedException>
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: 
    2016/10/11 08:09:45 [DEBUG] plugin: terraform.exe: -----------------------------------------------------
    2016/10/11 08:09:45 [DEBUG] root: eval: *terraform.EvalWriteState
    2016/10/11 08:09:45 [DEBUG] root: eval: *terraform.EvalApplyPost
    2016/10/11 08:09:45 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

    * aws_api_gateway_base_path_mapping.SeoWebsite: SerializationError: failed decoding REST JSON error response
    caused by: invalid character '<' looking for beginning of value
    2016/10/11 08:09:45 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

    * aws_api_gateway_base_path_mapping.SeoWebsite: SerializationError: failed decoding REST JSON error response
    caused by: invalid character '<' looking for beginning of value
    2016/10/11 08:09:45 [ERROR] root: eval: *terraform.EvalOpFilter, err: 1 error(s) occurred:

    * aws_api_gateway_base_path_mapping.SeoWebsite: SerializationError: failed decoding REST JSON error response
    caused by: invalid character '<' looking for beginning of value
    2016/10/11 08:09:45 [TRACE] [walkApply] Exiting eval tree: aws_api_gateway_base_path_mapping.SeoWebsite (destroy)
    2016/10/11 08:09:45 [DEBUG] vertex provider.aws (close), got dep: aws_api_gateway_base_path_mapping.SeoWebsite (destroy)
    2016/10/11 08:09:45 [DEBUG] vertex provider.aws (close), got dep: provider.aws
    2016/10/11 08:09:45 [DEBUG] vertex aws_api_gateway_base_path_mapping.SeoWebsite, got dep: aws_api_gateway_base_path_mapping.SeoWebsite (destroy)
    2016/10/11 08:09:45 [DEBUG] vertex provider.aws (close), got dep: aws_api_gateway_base_path_mapping.SeoWebsite
    2016/10/11 08:09:45 [DEBUG] vertex root, got dep: provider.aws (close)
    2016/10/11 08:09:45 [DEBUG] vertex root, got dep: var.environment
    2016/10/11 08:09:45 [TRACE] Preserving existing state lineage "6daa8d5e-7137-44db-8be5-c92ab857eb2c"
    2016/10/11 08:09:45 [TRACE] Preserving existing state lineage "6daa8d5e-7137-44db-8be5-c92ab857eb2c"
    2016/10/11 08:09:45 [DEBUG] plugin: waiting for all plugin processes to complete...
    2016/10/11 08:09:45 [DEBUG] plugin: C:\Program Files\Terraform\terraform.exe: plugin process exited

I also noticed that the api_id and state_name are empty in terraform.tfstate which seems wrong since the very first deployment succeeded.

            "aws_api_gateway_base_path_mapping.SeoWebsite": {
                "type": "aws_api_gateway_base_path_mapping",
                "depends_on": [
                    "aws_api_gateway_deployment.SeoWebsite",
                    "aws_api_gateway_domain_name.SeoWebsite",
                    "aws_api_gateway_rest_api.SeoWebsite"
                ],
                "primary": {
                    "id": "seo.domain.nz/",
                    "attributes": {
                        "api_id": "",
                        "base_path": "",
                        "domain_name": "seo.domain.nz",
                        "id": "seo.domain.nz/",
                        "stage_name": ""
                    },
                    "meta": {},
                    "tainted": false
                },
                "deposed": [],
                "provider": ""
            },

[richardc]

Have you tired specifying the base_path attribute as "(none)". This string is a special token for this api which indicates you want to map to the uri directly. https://docs.aws.amazon.com/apigateway/api-reference/link-relation/basepathmapping-by-base-path/#remarks

[BerndWessels]

@richardc wow, thanks, you are right.

@crumley This actually fixed the issue for me. At least I can now re-run terraform apply without getting any errors. The only change as mentioned was to have a base_path = "(none)" rather than not specifying the base_path.

@stack72 Maybe that can make it into the documentation. The docs say base_path is optional which is is not until the bug is fixed.

Still not sure if not having a base_path or having a base_path = "" should error, but for now I am happy with the workaround.

[chadgrant]

i got hit by this too, the resource should default to "(none)" if that really means "" or nil ... imho

[kylelaverty]

For anyone looking into this issue, the UI delete was also not working. I checked this week and the delete button in the UI is once again working. The terraform scripts started working after I added the base_path = "(none)" as per @BerndWessels.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.