Amplify-cli: Removing @searchable from a model fails to update stack

[poppybank]

Describe the bug

$ amplify --version 1.1.7

This looks a lot like #794 but since that is closed I'm opening a new bug since it could be different.

I am trying to remove @searchable from a model. All I did was delete @searchable and then tried to do amplify push to make the cloud changes but it fails.

A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:
1) Edit an existing graphql schema that has @searchable on one of its models to remove @searchable
2) run amplify push

Errors start showing up around GraphQLAPIKey

UPDATE_FAILED               GraphQLAPIKey                              AWS::AppSync::ApiKey        Tue Mar 05 2019 15:23:07 GMT-0800 (Pacific Standard Time) API key not found: da2-s7usndw5ybcn5i6zcrwvxlnwcu (Service: AWSAppSync; Status Code: 404; Error Code: NotFoundException; Request ID: a1b04564-3f9d-11e9-a1a8-a39b785176a3)
UPDATE_ROLLBACK_IN_PROGRESS spark-20190218123525-apispark-CR4WIE34F814 AWS::CloudFormation::Stack  Tue Mar 05 2019 15:23:09 GMT-0800 (Pacific Standard Time) The following resource(s) failed to update: [GraphQLAPIKey].                                                                                                              

Final error messages after rollback:

Following resources failed

Resource Name: arn:aws:appsync:us-west-2:665143778343:apis/mijd5doyovfplaohn26st7kk5e/apikeys/da2-s7usndw5ybcn5i6zcrwvxlnwcu (AWS::AppSync::ApiKey)
Event Type: update
Reason: API key not found: da2-s7usndw5ybcn5i6zcrwvxlnwcu (Service: AWSAppSync; Status Code: 404; Error Code: NotFoundException; Request ID: a1b04564-3f9d-11e9-a1a8-a39b785176a3)
URL: https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/arn%3Aaws%3Acloudformation%3Aus-west-2%3A665143778343%3Astack%2Fspark-20190218123525-apispark-CR4WIE34F814%2F0699df30-353f-11e9-a0a1-0a9edf17d014/events

Expected behavior
The command should complete and the model is no longer searchable and elastic search resources have been de-provisioned.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS 10.14.3
• Browser Safari 12.0.3
  
  
  
  • Amplify CLI version 1.1.7. Note this was originally happening in 1.1.0 but I installed 1.1.7 to see if it was fixed as #794 indicated.
  
  

Additional context
Add any other context about the problem here.

[mikeparisstuff]

Hey it looks like the API key was deleted by AppSync since it expired which caused CloudFormation to lose a reference. You can get around this by setting "APIKeyExpirationEpoch" to "-1" in the parameters.json file. See https://aws-amplify.github.io/docs/cli/graphql#apikeyexpirationepoch. After pushing and removing the API key, you can recreate a longer living key by setting the "APIKeyExpirationEpoch" value to the time in seconds since epoch that you would like it to expire.

[poppybank]

Thanks @mikeparisstuff Does the API Key need to be there and valid for that push to happen? I tried changing APIKeyExpirationEpoch to -1 and it is still failing, but for a different reason now. Much more vague.

Following resources failed
✖ An error occurred when pushing the resources to the cloud
Resource is not in the state stackUpdateComplete

Higher up I saw this:

UPDATE_FAILED Product AWS::CloudFormation::Stack Wed Mar 06 2019 19:25:24 GMT-0800 (Pacific Standard Time) Embedded stack arn:aws:cloudformation:us-west-2:665143778343:stack/spark-20190218123525-apispark-CR4WIE34F814-Product-U6M6JDNJHNDQ/179233a0-353f-11e9-99ab-0ad236afdc80 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: Export mijd5doyovfplaohn26st7kk5e:GetAtt:ProductTable:StreamArn cannot be deleted as it is in use by spark-20190218123525-apispark-CR4WIE34F814-SearchableStack-1XZOQKHE4QQ3O
⠧ Updating resources in the cloud. This may take a few minutes...

Followed by these, but I'm assuming they're failing because the issue above.

UPDATE_FAILED               S3File                                     AWS::CloudFormation::Stack Wed Mar 06 2019 19:25:35 GMT-0800 (Pacific Standard Time) Resource update cancelled                                     
UPDATE_ROLLBACK_IN_PROGRESS spark-20190218123525-apispark-CR4WIE34F814 AWS::CloudFormation::Stack Wed Mar 06 2019 19:25:36 GMT-0800 (Pacific Standard Time) The following resource(s) failed to update: [S3File, Product].
⠼ Updating resources in the cloud. This may take a few minutes...

UPDATE_FAILED               apispark             AWS::CloudFormation::Stack Wed Mar 06 2019 19:25:44 GMT-0800 (Pacific Standard Time) Embedded stack arn:aws:cloudformation:us-west-2:665143778343:stack/spark-20190218123525-apispark-CR4WIE34F814/0699df30-353f-11e9-a0a1-0a9edf17d014 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to update: [S3File, Product].
UPDATE_ROLLBACK_IN_PROGRESS spark-20190218123525 AWS::CloudFormation::Stack Wed Mar 06 2019 19:25:45 GMT-0800 (Pacific Standard Time) The following resource(s) failed to update: [apispark].                                                   

[mikeparisstuff]

@poppybank I am looking into this issue. The reason for the error is that the new stack is removing an export in one of the model stacks that is now unnecessary because searchable is being removied. The issue is that the searchable resources depend on the model stack (because the streaming depends on the model table) causing the model stack to get updated first. Since the existing searchable stack still depends on the export that is now gone, the update fails. You should be able to sidestep the issue by removing your API's SearchableStack using the AWS CloudFormation console and then running push again without the searchable directive. Meanwhile, I will continue investigating and try to figure out a solution to this moving forward.

[poppybank]

Thanks @mikeparisstuff, I'll give that a try.

[poppybank]

@mikeparisstuff, that did solve the problem for now (thanks!)
While I have been trying to get this working again, I did a couple of other changes to the schema that were ostensibly rolled back, but not completely and that started causing more issues. At one point I removed a model from the schema and tried to push. The push failed and rolled back, but the resolvers for that model were deleted on the back end and from that point on, pushes failed because of those missing resolvers. I probably could have added the resolvers manually on the backend, but in this case I just deleted the model again from my local schema and when I pushed this time, after removing the SearchableStack, everything worked out.

So in addition to looking at the execution order when removing @searchable, take a look at incomplete rollbacks (not un-deleting the resolvers) when removing models are part of a failed push.

[kstro21]

@mikeparisstuff I'm commenting here to describe my experience with the same issue in order to avoid opening a new one.

Describe the bug
I tried to remove the @searchable directive from a model, but it is not possible using amplify.

To Reproduce
Steps to reproduce the behavior:

1. Add @searchable to a model in the schema.graphql
2. Run amplify push
3. Remove @searchable from a model in the schema.graphql
4. >Export i198123kjaskdh912:GetAtt:TodoTable:StreamArn cannot be deleted as it is in use by amplifytest-20190308140619-apiamplifytest-Q5YW46Y7U507-SearchableStack-3Y7BNSAD6DKM

Expected behavior
I expect to be able to remove @searchable from a model.

Desktop (please complete the following information):

• amplify 1.1.7

Additional context
The problem is that the table's StreamArn is only exported if the @searchable directive is added to the model. So, removing it cause the exported value to be removed, but, the "AWS::CloudFormation::Stack" that creates the "AWS::Elasticsearch::Domain" "DependsOn" the model, thus the stack containing the model will be updated first and fails.

The solution is to always export the table's StreamArn, not only when the @searchable directive is used. This is already requested in #987

[rpostulart]

@poppybank I am looking into this issue. The reason for the error is that the new stack is removing an export in one of the model stacks that is now unnecessary because searchable is being removied. The issue is that the searchable resources depend on the model stack (because the streaming depends on the model table) causing the model stack to get updated first. Since the existing searchable stack still depends on the export that is now gone, the update fails. You should be able to sidestep the issue by removing your API's SearchableStack using the AWS CloudFormation console and then running push again without the searchable directive. Meanwhile, I will continue investigating and try to figure out a solution to this moving forward.

Is it possible to just delete a cloudformation stack manually from the console and then start a amplify push again?

[kstro21]

@rpostulart that won't work. If you do that, you will blow the entire stack and you will be forced to start from 0.

[artista7]

Hey @mikeparisstuff this issue is annoying, any updates?

[24jr]

Had same issue just updated with npm install -g @aws-amplify/cli and tried again and it worked. They must've just fixed it. It did take a while like idk 20 mins maybe for me updating but eventually worked