Amplify-cli: Even with No Changes, have 10 minutes of 'Updating resources in the cloud. This may take a few minutes...'

[leantide]

Enhancement

In a project with over 20 amplify resources, an 'amplify push' will have about 10 minutes of 'Updating resources in the cloud. This may take a few minutes...' even when there are No Changes in the backend resources to actually be pushed.

If there are a few backend changes, I see these being updated very quickly in the cloud and then I have the same 10-minute delay.

Similarly, an 'amplify publish' will have 10 minutes of 'Updating resources in the cloud. This may take a few minutes...' above the time it takes to build and publish the client.

I suspect the 10 minutes is associated with the exponential back off to avoid the Cloud Formation throttling limit with large concurrency. See https://github.com/aws-amplify/amplify-cli/issues/705.

Wondering if the 10 minute delay is expected for the exponential back off solution and if there is any other solution that may be more optimal.

[kaustavghosh06]

@leantide Yes, we're currently looking into enhancements for improving the deployment time.

[yuth]

@leantide what version of the CLI do you have?

We recently changed the way we poll the CFN for updates in CLI version 1.1.5. Could you share the poll output when there is no output for 10 minutes. I am interested in seeing what were the events that happen before and after the wait.

[leantide]

I am using @aws-amplify/cli 1.1.3. I will upgrade to 1.1.6 and re-try.

Where do I find the poll output?

Here is the CLI output on the terminal using v1.1.3. It doesn't say much. Just stayed 11 minutes on 'Updating resources in the cloud. This may take a few minutes... ' with a rotating busy indicator and then it said 'All resources are updated in the cloud'.

$ amplify push

Current Environment: dev

| Category | Resource name | Operation | Provider plugin |
| -------- | --------------- | --------- | ----------------- |
| Auth | cognito6e1470c4 | No Change | awscloudformation |
| Hosting | S3AndCloudFront | No Change | awscloudformation |
| Storage | authorization | No Change | awscloudformation |
| Storage | users | No Change | awscloudformation |
| Storage | settings | No Change | awscloudformation |
| Storage | membership | No Change | awscloudformation |
| Storage | details | No Change | awscloudformation |
| Storage | counters | No Change | awscloudformation |
| Storage | products | No Change | awscloudformation |
| Storage | releases | No Change | awscloudformation |
| Storage | features | No Change | awscloudformation |
| Storage | items | No Change | awscloudformation |
| Function | settings | No Change | awscloudformation |
| Function | membership | No Change | awscloudformation |
| Function | details | No Change | awscloudformation |
| Function | counters | No Change | awscloudformation |
| Function | products | No Change | awscloudformation |
| Function | nodes | No Change | awscloudformation |
| Api | settings | No Change | awscloudformation |
| Api | membership | No Change | awscloudformation |
| Api | details | No Change | awscloudformation |
| Api | counters | No Change | awscloudformation |
| Api | products | No Change | awscloudformation |
| Api | nodes | No Change | awscloudformation |
? Are you sure you want to continue? Yes

Updating resources in the cloud. This may take a few minutes...

✔ All resources are updated in the cloud

[leantide]

Exact same result using cli 1.1.6. With no changes to be pushed it spends 11 minutes on 'Updating resources in the cloud. This may take a few minutes...' before getting to 'All resources are updated in the cloud'.

[yuth]

If it didn't show anything on the screen, I don't think there was any updates to be done. Whats the size of your <project-root>/amplify/backend/ folder. It might be the time taken to upload these.

I will investigate further to see if we upload the content inside the backend folder even when there are no changes or is it the time taken just to generate the hash

[leantide]

See image. amplify/backend total size is 721.2 MB but 99.9% of that is in the functions subfolders, which seems to be mostly the function builds in the dist subfolder (which also includes previous builds), as well the src/node_modules that 'npm install' brings in for each function build. The membership function subfolder has about 210 MB, which is just a lot more previous builds being retained.

So I would have 6 unchanged functions at 7.8MB each which is about 47 MB.

[leantide]

I deleted all older function builds except the latest dated in each distribution. I did this in /amplify/backend as well as in /amplify/#current-cloud-backend.

$amplify push then felt 4 of my 6 functions were 'Updated' (maybe the latest build by date wasn't the one which it was using) and therefore rebuilt and pushed 4 new function distributions to the aws cloud. This only took 6 minutes, down from what previously may have taken 13 min.

A subsequent 'amplify push' once again showed 'No Changes' for all components, and took only 4 minutes to indicate 'All resources are updated in the cloud', with no terminal console statements, which previously was taking 11 minutes.

Given the process time is associated with the overall size of the amplify/backend folder, maybe it has to do with synchronizing from backend to #current-cloud-backend? Or is amplify actually processing older builds?

[leantide]

Some weird and circular stuff.

So I then delete the 4 remaining previous builds since 4 new builds were just built. 'amplify push' then determined that 2 functions were changed, 1 of the 4 I just deleted a previous build, and 1 of the remaining 2 functions I didn't touch. I clearly don't understand the logic of how amplify determines change.

I cancelled the push as I thought I would be smart and just remove the latest build from the 2 functions that amplify now felt were changed and I got this error:

ENOENT: no such file or directory, lstat 'project-path/amplify/backend/function/products/dist/products-1551241782-latest-build.zip'

So I restored the 2 latest builds just removed, and amplify rebuilt them and pushed them. A subsequent push saw No Changes as expected.

[yuth]

I think I have narrowed down the problem. When a push is performed, irrespective of change/no change CLI zips the amplify/backend directory and saves it in S3. This is done to support multienv. To solve the slow push we should fix the following

1. Don't push anything if the resources have not changed
2. Delete older version of function builds when a new build is generated
3. When creating #current-cloud-backend.zip ignore the build directories from AppSync and Function resources.

[leantide]

With your item 1. in place, if no backend changes have been made but client side changes are made, just to affirm that an 'amplify publish' would not make any backend push but would still build and publish the client.

[leantide]

For your item 3., there is still a substantial size to my function/[name]/src/node_modules subdirectories. Since these are reconstructed by an npm install, I don't store them in my code repo. Would these node_modules also be candidates to ignore along with the build/dist directories?