Amplify-cli: Amplify CLI 4.30 fails to update backend stack (fails to upload new cloudformation files and hence doesn't execute them)

[pepso]

Describe the bug
Yesterday, using 4.29.8 (as part of Amplify console), amplify CLI uploads new and changed files (cloud formation I assume) and executes cloud update. Today, with 4.30, cloud formation/files are not being uploaded and hence backend fails to update.

Example logs with amplify-cli 4.29.8
2020-10-17T13:53:19.373Z [INFO]: Successfully added environment from your project
2020-10-17T13:53:19.672Z [INFO]: # Initializing existing Amplify environment: dev (amplify init)
2020-10-17T13:53:20.668Z [INFO]: Note: It is recommended to run this command from the root of your app directory
2020-10-17T13:53:24.772Z [WARNING]: - Initializing your environment: dev
2020-10-17T13:53:24.794Z [WARNING]: (node:1993) ExperimentalWarning: The fs.promises API is experimental
2020-10-17T13:53:25.401Z [WARNING]: ✔ Initialized provider successfully.
2020-10-17T13:53:59.267Z [WARNING]: - Uploading files...
2020-10-17T13:54:00.101Z [WARNING]: - Uploading files...
2020-10-17T13:54:01.179Z [WARNING]: - Uploading files...
2020-10-17T13:54:02.420Z [WARNING]: - Uploading files...
2020-10-17T13:54:03.413Z [WARNING]: - Uploading files...
2020-10-17T13:54:03.651Z [WARNING]: - Uploading files...
2020-10-17T13:54:07.285Z [WARNING]: - Uploading files...
2020-10-17T13:54:07.762Z [WARNING]: - Updating resources in the cloud. This may take a few minutes...

Example logs with amplify-cli 4.30:
2020-10-18T10:09:06.712Z [INFO]: Successfully added environment from your project
2020-10-18T10:09:06.874Z [INFO]: # Initializing existing Amplify environment: dev (amplify init)
2020-10-18T10:09:07.864Z [INFO]: Note: It is recommended to run this command from the root of your app directory
2020-10-18T10:09:11.805Z [WARNING]: - Initializing your environment: dev
2020-10-18T10:09:11.825Z [WARNING]: (node:1996) ExperimentalWarning: The fs.promises API is experimental
2020-10-18T10:09:12.353Z [WARNING]: ✔ Initialized provider successfully.
2020-10-18T10:09:12.545Z [INFO]: Initialized your environment successfully.
2020-10-18T10:09:12.545Z [INFO]: Your project has been successfully initialized and connected to the cloud!

Amplify CLI Version
4.30

To Reproduce
Run amplify console pipeline with tag "latest" using 4.30 and see no file uploads nor cloud updates are executed for backend changes.
Run amplify console pipeline with tag "4.29.8" for the same commit and see file uploads and cloud updates are executed normally.

Expected behavior
Existing amplify projects continue to work and upload backend changes to the cloud.

Additional context
Workaround: use 4.29.8 until there is fixed applied or instructions on how to fix the issue going forward

[akshbhu]

Hi @pepso

We have suppressed printing multiple uploading files . Is there any issue you are facing with this ?
For the backend you can check your S3 deployment bucket.

Let me know if this answers your question?

[pepso]

Hi @akshbhu

In my project, the time it takes to upload cf-files ("✔ Initialized provider successfully. " to "Updating resources in the cloud. This may take a few minutes...") takes consistently 40-45 seconds with 4.29.8, and it seems new CLI 4.30 gets through that in few seconds. Basically doesn't upload the files and hence doesn't apply them either?

Maybe a more relevant point is that a new database, API, and associated lambda I created on Sunday morning didn't get pushed to my cloud environment. I narrowed the issue down with a small code change to existing lambda function and can confirm it didn't get updated either. This is when I reverted to 4.29.8 as a workaround and raised this issue.

Something smells fishy in that part of the build/push as all my (amplify console) runs with CLI 4.30 are consistent 4 minutes. With reverted CLI 4.29.8 they take around 6 minutes. I like the idea you guys managed to cut 2 minutes of build/push time for my project but in this case, this speed-up broke cloud updates being applied.

Am I the only one who is experiencing this?

[evuori]

Am I the only one who is experiencing this?

You are not alone Sir.

Updated this morning to 4.30. After tweaking graphql schema and issuing "amplify status" CLI says No Change.

I am sorry to say but at this moment I don't have more info around. (I had to pull down quickly back to 4.29.8)

[akshbhu]

Hi @ pepso

I tried reproducing the behavior with 4.30 and all my schema and resolvers are getting updated. Ill try with projects created with previous versions and update with latest(4.30) one. Can you tell me changes you made so that I can reproduce this on my end?

Hi @evuori

This seems like different issue as it is not detecting changes so the push wont happen if no changes are detected. Can you tell me more on this issue as I am not able to reproduce this with V4.30

[evuori]

Hey @akshbhu, just tried again with 4.30 and can't reproduce "No Change" issue.
(really don't know what was going on but for sure it was not detecting my schema changes)

[SwaySway]

@pepso
What categories do you have enabled in your project? Is it possible for you to send over the amplify/ directory to [email protected]? We can look into reproducing the problem.

@evuori
This seems to be different that the OP's issue. Could you open another issue for this? Also explain the change done to the schema which would work on a previous version and not on 4.30.

[mikedizon]

@SwaySway seeing the same behavior. Rolled back a version and everything worked fine again.

[pepso]

@SwaySway I'm using API, Auth, Function, Storage and Hosting categories in my project. Unfortunately, I am not able to send over the amplify backend for you.

I just tested 4.29.8, 4.30.0 and the new 4.31.0 versions with a simple whitespace change to my lambda function:

Amazon Linux:2 base image
4.29.8 Amplify Console Build time: 6m33s
4.30.0 Amplify Console Build time: 4m48s
4.31.0 Amplify Console Build time: 4m54s

Using amplify console, I executed the simple change build with 4.30 first, then with 4.31, and using 4.29 as a last one. Meanwhile, I observed amplify deployment cfn&build bucket in S3 and can confirm 4.29.8 is the only one which did file uploads (*build.zip, *cloudformation-template.json "Last modified" updated). At least this should narrow down the scope of the problem further.

It's like my amplify-cli doesn't detect the changes (or push them) when above 4.29.8.

Can it be what I am experiencing is related to this: https://github.com/aws-amplify/amplify-cli/pull/5546

When I have a moment I'm going to look into my build scripts and identify which flags I'm using. If I find something I'll come back to this issue.

[SwaySway]

@mikedizon
A few questions here

1. What categories did you have created in your amplify project.
2. Which version did you roll back to? To better track where the potential bug might be.

[mikedizon]

@SwaySway

Categories:

• Api
• Function
• Auth
• Storage

Api Version:

• 4.29.8

[mikedizon]

@SwaySway i should add that I have extended the amplifyPush.sh file

https://github.com/aws-amplify/amplify-console/blob/2aafa34c414bacbd7f83913f34eb9bd2b1c13068/scripts/amplifyPush.sh

[pepso]

@mikedizon that is a good call. I use custom amplifyPush.sh script to store Auth params as part of the deployment.

@SwaySway maybe relevant as that is the only script which defines any flags in my deployment process.

[BeaveArony]

I can confirm that when using the amplifyPush.sh script the line

amplify init --amplify ${AMPLIFY} --providers ${PROVIDERS} --codegen ${CODEGEN} --yes

used to also push after doing the init stuff. Now with 4.30.0 and above it will only init, but not push anymore.

This is a breaking change that I have missed somehow.