Amplify-cli: Error when pushing to existing environment: API key not found

[mwarger]

Describe the bug
When I push to an existing environment to update the api, I get an error saying "API key not found"

Amplify CLI Version
amplify -v
3.17.0
To Reproduce
I'm not sure what led to this.

Expected behavior
I would expect it to work.

[davidbiller]

So did you add the line?:
https://aws-amplify.github.io/docs/cli-toolchain/graphql#apikeyexpirationepoch
{ "APIKeyExpirationEpoch": "-1" }

if yes, but you dont use API Key in your API, than delete this line.

[attilah]

@mwarger could you please confirm that the API key has expired and was deleted by the Service is the case for you?

If yes, these steps help to let CF know about the missing Api Key and get it into a consistent state.

After that if you add: { "APIKeyExpirationEpoch": "-1" } to parameters.json, you can disable the API Key creation, or just do a whitespace change in the schema and push the project as it is, it will recreate the API key with the configured API key expiration (7 days by default), what you can update with amplify api update.

There is an open PR around API key management fixes: https://github.com/aws-amplify/amplify-cli/pull/2678

[mwarger]

I deleted the line, and it failed. I tried to create a new environment, and it failed. @attilah @kaustavghosh06 This is trying my patience. I will get back to this with more logs later.

[chadkouse]

@attilah

If yes, these steps help to let CF know about the missing Api Key and get it into a consistent state.

Could you explain which steps you're referring to? Having the same problem.

[cottageGrove]

stuck in the same place. tried { "APIKeyExpirationEpoch": "-1" }. there should be something in the console that allows you to manually change the API Key.

[chadkouse]

Ok well this worked for me - I went into CloudFormation and manually edited the stack (the root stack) and set CreateApiKey to 0 and saved it.. This allowed the stack to get into a consistent state. Then I went back to amplify cli and did the following
amplify env checkout prod (this may not be necessary?)
amplify env pull (this may not be necessary?)
amplify api update -> went ahead and reconfigured my API, specifically telling it to use an API Key (my app uses it)
after that, amplify push worked and it created a new API key (although it seemed to not honor my the 365 day expiration date I told it to use during amplify api update) -- but I'm at least back up and running.

I opened a PR for the expiration date issue I mentioned

[cottageGrove]

Hey folks I just updated amplify to the latest version and added { "APIKeyExpirationEpoch": "-1" } and it worked

[mwarger]

This just magically worked the next time I tried to deploy it. I dunno.

[davekiss]

@mwarger @attilah can you please reopen this issue? This randomly hit my production environment as well and I'm currently sitting in a broken state with no changes on my end. My API Key was set up originally to expire 180 days out, it certainly hasn't been that long, so it's extremely concerning that this issue just cropped up on its own.

What steps need to be taken to "get CF into a consistent state?"

@kaustavghosh06 might you know the answer to this?

[chadkouse]

@davekiss just curious - is API Key your primary auth provider or a secondary one?

[davekiss]

@chadkouse it's secondary, am also using IAM auth and userPools (User Pools is primary)

[attilah]

These manual steps can help to recover into a stable state from CFN point of view, from that point APIKeyExpirationEpoch can be used as documented. CreateAPIKey usage is fixed and will be included in the next release.

The steps:
There is a solution to recover from this by doing a change set deployment on the nested stack of the API.

Here are the steps:

• Go to CloudFormation Console and turn on nested stacks
• Locate and select the API nested stack which failed to update due to the removed API key, it should have the same prefix as the root stack followed by the API name
• Click on the Update button
• Select the "Update nested stack" option and click "Update stack"
• Select the "Use current template" option and click next
• Set APIKeyExpirationEpoch to -1
• Set CreateAPIKey to 0
• Click "Next"
• Click "Next" on the "Configure stack options screen"
• Check the acknowledgement checkbox(es) in the Capabilities section

After the Changes are calculated you should see a "Remove" action for the "GraphQLAPIKey", if that's the case, proceed.

• Click "Update stack"

The API now should be in an updatable state.

[chadkouse]

@davekiss see this ticket and this PR (merged - but i do not believe released yet) for explanation of why your api key's expiration date got ignored / reset to the default (7 days)

[davekiss]

@attilah @chadkouse thank you for your help, this solves the issue. I am sure bugs like this are hard to watch unfold. It brings up a bigger question for me as two days downtime in production due to bugs like this can literally be the end my business –– are we sure this project is production ready?

Also, honest question –– what is the best way to expedite support in cases like this to ensure a timely fix? I am guessing that internally at AWS, the folks writing the CLI code are the ones that would need to provide a resolution to issues like this, but the expectation that you can provide speedy support _and_ write production code along with all of your other responsibilities seems unfeasible. I understand this is semi-OSS but because it is tied to paid AWS services I'd like to know what is possible.

I don't mean to sound entitled but as mentioned above, bringing production systems down to a halt just isn't an option if I want to stay in business. Is my best option really to click around and precariously tweak settings in the console to try and find the bug in code that I didn't write?

[BrittonWinterrose]

Encountered a similar issue after manually deleting the API keys via the AppSync view of the AWS Managment Console.

@attilah's troubleshooting steps worked perfect. Thanks!