Terraform-provider-aws: dynamodb-local plan / apply fails on UnknownOperationException for DescribeTimeToLive

[bglick]

Terraform Version

0.9.11 (also tested 0.9.8)

Affected Resource(s)

Please list the resources as a list, for example:

• dynamodb_table

Terraform Configuration Files

# Configure the AWS Provider
provider "aws" {
  region     = "us-east-1"
  endpoints {
    dynamodb = "http://localhost:8000"
  }
}

resource "aws_dynamodb_table" "delete_me_table" {
  name           = "dev-delete-me"
  read_capacity  = 5
  write_capacity = 5
  hash_key       = "organization_uuid"
  range_key      = "created_at"

  attribute {
    name = "organization_uuid"
    type = "S"
  }

  attribute {
    name = "created_at"
    type = "N"
  }
}

Debug Output

https://gist.github.com/bglick/35899492e8ec5f9ab8908f5f24d863fa

Error is around line 50

Expected Behavior

Running plan after applying the first time should produce no changes

Actual Behavior

Error is generated, UnknownOperationException which is really dynamo returning a 400 for the call to dynamodb/DescribeTimeToLive

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. terraform plan

Important Factoids

• Running against the latest version of dynamodb-local

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• hashicorp/terraform#14104 appears to be the PR where the feature was added that is causing the issue

[vancluever]

It goes farther than DescribeTimeToLive, unfortunately (which isn't apparently supported by DynamoDB local). ListTagsOfResource seems to be having the same issue:

2017/07/06 15:53:54 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/07/06 15:53:54 [DEBUG] plugin: terraform: aws-provider (internal) 2017/07/06 15:53:54 [DEBUG] [aws-sdk-go] DEBUG: Request dynamodb/ListTagsOfResource Details:
2017/07/06 15:53:54 [DEBUG] plugin: terraform: ---[ REQUEST POST-SIGN ]-----------------------------
2017/07/06 15:53:54 [DEBUG] plugin: terraform: POST // HTTP/1.1
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Host: localhost:32771
2017/07/06 15:53:54 [DEBUG] plugin: terraform: User-Agent: aws-sdk-go/1.8.13 (go1.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.9.4
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Content-Length: 66
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Accept-Encoding: identity
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Authorization: AWS4-HMAC-SHA256 Credential=foo/20170706/ca-central-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=xxxx
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Content-Type: application/x-amz-json-1.0
2017/07/06 15:53:54 [DEBUG] plugin: terraform: X-Amz-Date: 20170706T225354Z
2017/07/06 15:53:54 [DEBUG] plugin: terraform: X-Amz-Target: DynamoDB_20120810.ListTagsOfResource
2017/07/06 15:53:54 [DEBUG] plugin: terraform: 
2017/07/06 15:53:54 [DEBUG] plugin: terraform: {"ResourceArn":"arn:aws:dynamodb:ddblocal:000000000000:table/foo"}
2017/07/06 15:53:54 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/07/06 15:53:54 [DEBUG] plugin: terraform: aws-provider (internal) 2017/07/06 15:53:54 [DEBUG] [aws-sdk-go] DEBUG: Response dynamodb/ListTagsOfResource Details:
2017/07/06 15:53:54 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/07/06 15:53:54 [DEBUG] plugin: terraform: HTTP/1.1 400 Bad Request
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Connection: close
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Content-Type: application/x-amz-json-1.0
2017/07/06 15:53:54 [DEBUG] plugin: terraform: Server: Jetty(8.1.12.v20130726)
2017/07/06 15:53:54 [DEBUG] plugin: terraform: X-Amzn-Requestid: xxxx
2017/07/06 15:53:54 [DEBUG] plugin: terraform: 
2017/07/06 15:53:54 [DEBUG] plugin: terraform: {"__type":"com.amazonaws.dynamodb.v20120810#UnknownOperationException","message":"An unknown operation was requested."}
2017/07/06 15:53:54 [DEBUG] plugin: terraform: -----------------------------------------------------

Probably means that several calls are going to need to be skipped in the event of DynamoDB local (and it will have to be detected somehow).

[bglick]

@vancluever Detecting the difference seems to be the big issue here. I can't identify a non-hack call that could be made to figure it out.

Perhaps a good path is to add a local_mode option in the configuration and leave it to the user to explicitly set it on their test environments.

[vancluever]

Yeah, I agree. I don't see that being a really invasive change either and it will be easy to check against when making calls.

[bglick]

I'm not a Go developer, so I won't be able to write the patch, but if it helps, I could run through the current DynamoAPI and identify all of the calls that currently fail and report back the list. Just let me know.

[Ninir]

Hey folks,

To resume, this issue is happening ONLY when using a local DynamoDB, with new API Calls unsupported by the local API (describe TTL, probably Accelerator, tags, ...).
Using the provided config without the provider one works as expected.

At this point, the only thing we could do is to catch the UnknownOperationException..

@bglick could you check using the latest version of the local DynamoDB? Also, what do you mean by Perhaps a good path is to add a local_mode option in the configuration and leave it to the user to explicitly set it on their test environments.? not sure to get the point here.

Thanks!

[stevenao]

It looks me it is more of a bug from the AWS side that they miss to take care in DynamoDB local.

[bglick]

@Ninir I'll try to get to documenting the list of calls against the most recent this week.

As far as Perhaps a good path is to add a local_mode option in the configuration and leave it to the user to explicitly set it on their test environments.. My point was that I think it will be very hard to detect whether we are talking to the "real" Dynamodb or the local version. If you're using the local in a dev environment, you should be able to set a flag like local_mode to true directly in your .tf file.

This way the vast majority of executions that are not using the local version don't have to have some sort of hack in the code to try to figure out if they're touching the local server before executing (which could potentially introduce all sorts of bugs).

[nterry]

@stevenao Agreed, the ttl should be a stub, at least. The fact that we have to patch around this is absurd

[stevenao]

Someone should press AWS to handle those operations in their dynamodb local implementation. They can always just do no-op and let the api through. It is not hard to add. As a matter of fact, one can create a reverse proxy and handle the new operations api, but forward the known ones to dynamodb local instance.

[ben-bourdin451]

@stevenao I had the same thought about a reverse proxy and found a potential candidate in this repo however just from looking at the code it seems that it doesn't do what we need it to.

I'm wondering how difficult this would be to implement, or if it's even worth it 🤔 That is of course if no-one else has already done this...

[stevenao]

@ben-bourdin451 Local-Stack. That stuff looks a little overwhelming to me. Anyways, I put together a POC into a new repo (https://github.com/stevenao/dynamodb-local-mock-unsupported-api). It is currently mocking out the Tagging and TTL api. It is simply running a nodejs api server in front of the dynamodb-local api server to provide mocks to the unsupported api's.

[ben-bourdin451]

@stevenao Thanks for that! I briefly tested this today and had some issues but I will comment on your repo 😄

[osama-afifi]

Pull request on localstack fixing this issue https://github.com/localstack/localstack/pull/599

[stefansundin]

I just created a PR to work around the issue reading tags (https://github.com/terraform-providers/terraform-provider-aws/pull/6149). I wasn't able to reproduce the issue with TTL. Maybe they fixed it with the more recent dynamodb-local versions? Can anyone confirm this?

It looks like dynamodb-local doesn't support setting TTLs (persistent diff), but Terraform does not seem to crash for me.

UPDATE: Found this in the release notes in the zip file (from here).

2017-04-13 (1.11.119)

• Add TTL implementation
• Update aws libs to 1.11.119

But this date is earlier than when this issue was opened. Was an old version used perhaps?

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!