Amplify-console: Deploy fails with "[ERROR]: Failed to deploy"

[desertmonkey]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Start a build on our master branch
2. See error during the Deploy step.

Expected behavior
The build should succeed, as it has every other time before today.

Additional context
We have a stage branch and a master branch in Amplify. We merge GitHub release branches into the stage branch first to verify it all looks good. The builds on stage work flawlessly. Then we merge that same release branch into the master branch and the build fails to deploy.

I thought this could be related to this issue but I reduced the size of our build by a couple hundred MBs, and it still fails to deploy.

[cawleyflower]

@desertmonkey and I have been solving this issue
This deploy bug has been persistent in our Amplify app for a while now. I first noticed when trying to set up a new branch with Amplify, and were recieving this same build and deploy error consistently. It had never affected the master (production) branch until today.

I tried a number of fixes to get this working. We removed some duplicate binary files, which cut the build artifact down by half. This alone didn't fix the deploy error.

I finally have a fix in to get the master branch up and running. We are on a old version of Hugo, and our build requires an extended version to build. I moved the wget and installation Build Commands into the Prebuild, and the app started to deploy.

The bug still persists, and could have been quite costly.

[swaminator]

@cawleyflower thanks for providing the update. We have a known issue with deployment sizes. We'll update you when the fix is rolled out to all regions.

[kununguaq]

AWS amplify console completes provision & build stages, but fails at deploy without giving a reason... The last 3 lines of my deployment log:

2019-09-23T22:56:23 [INFO]: Updating Edge config
2019-09-23T22:56:23 [ERROR]: Failed to deploy
2019-09-23T22:56:23 [INFO]: Deployment finished.

The crazy thing is i can't even re-deploy a previously stable build. What's up ?
My App ARN: arn:aws:amplify:eu-west-1:352247527096:apps/d1p38jyf52tcwc

[Weicocu]

Same exact issue. My project is pretty simple, and my zipped build artifact is exactly 8000 KB, which seems suspicious. The last build that deployed successfully has a 7998 KB build artifact.

Looks similar to this issue, but they said something about a 1 GB build artifact limit. So my 8MB limit seems crazy small!

[swaminator]

146

[niallmc]

I am having the same issue:

2019-09-24T00:04:49 [INFO]: Updating Edge config
2019-09-24T00:04:49 [ERROR]: Failed to deploy
2019-09-24T00:04:49 [INFO]: Deployment finished.

[swaminator]

Can you try now? A fix has been rolled out.

[Weicocu]

Fix worked for me, thank you!!

[suzuky43]

I have same issue. So my issue didn't solve yet. I use ap-northeast-1 region.
Could you tell me the correct version of Amplify Cli?
My app size is about 500MB. It's smaller than 1GB.
So, if amplify were fixed already, my app would be deployed...

[cawleyflower]

@swaminator We saw the issue go away, but now that our build size has reached 500 MB, we are failing to deploy.

[cawleyflower]

@swaminator is there any way we can have our deploy size limit increased as it states here #41 ?

Our ARN is here: arn:aws:amplify:us-west-2:054500231008:apps/d1ev2ep2azk4pn

[thys-unomena]

Can you try now? A fix has been rolled out.

Thank you very much!

Has this been rolled out for the region 'us-west-2' also?
I'm still seeing this issue. Even now that our build artifact size was reduced from 500 MB to 26 MB

[swaminator]

@thys-unomena can you share your appid?

[cawleyflower]

@swaminator @thys-unomena is referring to the same app with ARN: arn:aws:amplify:us-west-2:xxxxxxxxxxxxxx:apps/d1ev2ep2azk4pn

[thys-unomena]

@swaminator @thys-unomena is referring to the same app with ARN: arn:aws:amplify:us-west-2:054500231008:apps/d1ev2ep2azk4pn

Hi @swaminator
Did you do something on your side? About 10 hours ago the builds started deploying successfully.

If it was something that you did, can you please provide some kind of an explanation?
And how can we prevent this from happening again?

[rickiesmooth]

I'm having the same problem since I've added cypress E2E tests

[JenniferMcIntyre-xyz]

Was this solved in #101?

I'm still getting the same error on my staging and prod environments. I'm going to take out some of the assets that I no longer use to see if it makes a significant change.

[pcoleman-pixafy]

@swaminator this is happening on the staging-mx branch of my build as well:

ARN: arn:aws:amplify:us-east-2:550189166758:apps/d1wxird61210hn

My team and I have 3 other branches that deploy fine without issue.

[dzincone]

Having the same issue on my 'dev' branch

ARN: arn:aws:amplify:us-east-2:392911942550:apps/dggeq6y7mtf3z

[rickiesmooth]

does it help putting your ARN number here?

[JenniferMcIntyre-xyz]

It was a cache size error for me. I solved it by removing the branch and re-adding it.

[dzincone]

My issue was solved by correcting the build path for the build step. Was defaulted to / and needed it at /build

[rickiesmooth]

weird, I re-added my branch (by removing and adding the app?) and as far as I can see my build path is correctly setup (for me it's web-build). Would be great if someone from the amplify team could advise.

[rickiesmooth]

@swaminator maybe you have an idea to reset the cache or something? I'm having this issue for a while now since I've added cypress tests.

[litwicki]

@rickiesmooth forgive the obvious question, but if you disable Cyprus are you able to build?

[rickiesmooth]

@litwicki thanks for getting back to me, yes I've tried to disable all test but that didnt help unfortunately. I did notice that when I try to amplify publish it locally it hangs on uploading my build artifacts

Index document is currently set as:
  index.html

⠏ Uploading files...

[bordeo]

@swaminator
I have the same problem with my app d1ll0ojvkzv9e7

Could you help me?

[Zelig880]

Hi guys,

I have the same issue. The App id is: d10w1ixpj82f3d

To provide you more details:

• I have deployed over 70 branches with no prob.
• Old branch still work
• I was doing some work and a branch failed
• I have created another branch ( with just variable name change) and it also fails

[nimacks]

We're actively investigating issues related to builds failing.

[nimacks]

I have the same issue. The App id is: d10w1ixpj82f3d
@Zelig880 , can you provide the region your app is in?

[nimacks]

I have the same problem with my app d1ll0ojvkzv9e7
@bordeo , can you provide the region your app is in?

[nimacks]

@rickiesmooth , can you provide your appId and region?

[bordeo]

@nimacks eu-west-1

[Zelig880]

Us West (Oregon)

On Fri, 8 Nov 2019, 5:33 pm Samuel Mensah, [e-mail] wrote:

I have the same issue. The App id is: d10w1ixpj82f3d
@Zelig880 https://github.com/Zelig880 , can you provide the region your
app is in?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws-amplify/amplify-console/issues/115?email_source=notifications&email_token=ACE7HAR65P7CEQBLAHNXNYLQSWPHXA5CNFSM4IWULLI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDS2FEY#issuecomment-551920275,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACE7HATEVQL7IZB5CD54OZDQSWPHXANCNFSM4IWULLIQ
.

[nimacks]

@bordeo, it looks like your build is failing here are some logs :

2019-11-08T11:16:17.817Z [INFO]: $ gatsby build 
2019-11-08T11:16:19.689Z [INFO]: success open and validate gatsby-configs — 0.071 
2019-11-08T11:16:20.353Z [INFO]: success load plugins — 0.601 
2019-11-08T11:16:20.361Z [INFO]: success onPreInit — 0.008 
2019-11-08T11:16:22.069Z [INFO]: success delete html and css files from previous builds — 1.708 
2019-11-08T11:16:22.086Z [INFO]: success initialize cache — 0.017 
2019-11-08T11:16:22.118Z [INFO]: success copy gatsby files — 0.031 
2019-11-08T11:16:22.136Z [WARNING]: error 
                                    "bucketName" is a required option for gatsby-plugin-s3 
                                    See docs here - https://github.com/jariz/gatsby-plugin-s3 
             See our docs page for more info on this error: https://gatsby.dev/issue-how-to 
2019-11-08T11:16:22.210Z [WARNING]: error Command failed with exit code 1. 
2019-11-08T11:16:22.210Z [INFO]: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 
2019-11-08T11:16:22.218Z [ERROR]: !!! Build failed 

[bordeo]

@nimacks this is the last build. I had errore on build 69/70. I don’t remember. The error are on deploy step

[bordeo]

@nimacks the build on deploy error is 64

[Zelig880]

@nimacks I do apologies, I have provided you the wrong region above.

My region is US East (N. Virginia).

I have just tried to deploy one of the failing branch again, and it is still failing. I will try now to create a new Branch, just in case the problem has been solved elsewhere.

Please keep us updated on the progress and the cause behind this issue, and it is affecting project delivery!

[nimacks]

We have identified an issue in which customers with a very large set of files are unable to deploy their sites. We are actively prioritizing a fix for this. In the meantime, customers can delete unused branches in their app as a workaround.

[rickiesmooth]

I'm still seeing this issue and I believe I have only one branch? My appId is dmm4bu2tim2c and the region is eu-central-1, hope it helps!

[rkul]

We have just 9 PRs opened and the deployments fail randomly now with the same message as above:

2019-11-18T13:19:48 [INFO]: Updating Edge config
2019-11-18T13:19:48 [ERROR]: Failed to deploy
2019-11-18T13:19:48 [INFO]: Deployment finished.

Application ARN is arn:aws:amplify:us-east-1:044883579400:apps/d2lwxktxnpxork

@nimacks could you provide any ETA or workaround? Our main frontend application in production is affected.

[Joycehao19]

Hello @rkul , you could try:

1. If you have any branches that aren't being actively used, you can disconnect those branches which should be able to unblock you.
2. Connect a new app.

If you still have this issue, please let us know.

[Joycehao19]

Hello @rickiesmooth , I checked your build on Nov 17, the reason of deploy failed is:
(node:49) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '/codebuild/output/src366734173/src/home/../testConfig.json'

Could you double check if the file or directory exist?

[rickiesmooth]

@Joycehao19 thanks for looking into this! I believe this file was used for cypress, but that's not on master anymore. So I'm not sure which version of my code it's trying to build there..

[kkesley-shine]

we have the same problem currently.. our app id is d3rfa777e3c1ou in ap-southeast-2.

don't know if this is relevant,

but we noticed that the error comes when we changed the cypress version in yarn lock. If we use 3.4.1, the tests are running. (sometimes it freezes but it's running). When we upgraded our cypress version to 3.6.1 in package.json, it fixes the freeze.

But when we changed our yarn.lock to include that 3.6.1, it doesn't even run the test. These are the last log from test tab

cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"

and then it goes straight to deployment

2019-11-21T04:40:01 [INFO]: Starting Deployment
2019-11-21T04:40:01 [ERROR]: Failed to deploy
2019-11-21T04:40:01 [INFO]: Deployment finished.

Update

I can confirm that the cypress version in yarn.lock is causing it.

Because when I rollback to:

[email protected]:
  version "3.4.1"
  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.4.1.tgz#ca2e4e9864679da686c6a6189603efd409664c30"
  integrity sha512-1HBS7t9XXzkt6QHbwfirWYty8vzxNMawGj1yI+Fu6C3/VZJ8UtUngMW6layqwYZzLTZV8tiDpdCNBypn78V4Dg==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
    arch "2.1.1"
    bluebird "3.5.0"
    cachedir "1.3.0"
    chalk "2.4.2"
    check-more-types "2.24.0"
    commander "2.15.1"
    common-tags "1.8.0"
    debug "3.2.6"
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
    fs-extra "5.0.0"
    getos "3.1.1"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    url "0.11.0"
    yauzl "2.10.0"

it works.

Previously, we used:

[email protected]:
  version "3.6.1"
  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.6.1.tgz#4420957923879f60b7a5146ccbf81841a149b653"
  integrity sha512-6n0oqENdz/oQ7EJ6IgESNb2M7Bo/70qX9jSJsAziJTC3kICfEMmJUlrAnP9bn+ut24MlXQST5nRXhUP5nRIx6A==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
    "@types/sizzle" "2.3.2"
    arch "2.1.1"
    bluebird "3.5.0"
    cachedir "1.3.0"
    chalk "2.4.2"
    check-more-types "2.24.0"
    commander "2.15.1"
    common-tags "1.8.0"
    debug "3.2.6"
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
    fs-extra "5.0.0"
    getos "3.1.1"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    untildify "3.0.3"
    url "0.11.0"
    yauzl "2.10.0"

note: that we're still using 3.6.1 in package.json. And we can confirm amplify is using 3.6.1 when running the test.

[nunsie]

Experiencing the exact same issue, it seems the build step isn't respecting the directory configured in: frontend > artifacts > baseDirectory and caching the entire directory including src and node_modules resulting in a cache size of over 1.2GB, my build folder size should only be 96MB

[nunsie]

removing tests step resolves the issue..

[nadinagray]

Hi -- experiencing the exact same issue. Have tried: removing branches, removing PR previews, creating a new app. @nimacks any suggestions or ideas for a workaround?

Region: us-west-2
App id: d1kf7vuhz1gmur

[kahdojay]

@nadinagerlach we took a look and it appears you are running in a limit of the number of files in your deploy artifacts. This is a known issue for which we're working on a fix. Since it's a per-app limit, one workaround would be to create a new app in the Amplify Console for a new branch. Please give it a try and let us know if it works.

[ascorbic]

@kahdojay Does this mean we can't use PR previews? I'm trying to deploy a moderately-large Gatsby site, which has around 15000 files, and it deployed fine initially but since adding PR previews deploys are failing with this error. Is there a timeline for a fix? If not we'll have to move away from Amplify entirely.

[rickiesmooth]

we have the same problem currently.. our app id is d3rfa777e3c1ou in ap-southeast-2.

don't know if this is relevant,

but we noticed that the error comes when we changed the cypress version in yarn lock. If we use 3.4.1, the tests are running. (sometimes it freezes but it's running). When we upgraded our cypress version to 3.6.1 in package.json, it fixes the freeze.

But when we changed our yarn.lock to include that 3.6.1, it doesn't even run the test. These are the last log from test tab

cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"

and then it goes straight to deployment

2019-11-21T04:40:01 [INFO]: Starting Deployment
2019-11-21T04:40:01 [ERROR]: Failed to deploy
2019-11-21T04:40:01 [INFO]: Deployment finished.

_Update_

I can confirm that the cypress version in yarn.lock is causing it.

Because when I rollback to:

[email protected]:
  version "3.4.1"
  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.4.1.tgz#ca2e4e9864679da686c6a6189603efd409664c30"
  integrity sha512-1HBS7t9XXzkt6QHbwfirWYty8vzxNMawGj1yI+Fu6C3/VZJ8UtUngMW6layqwYZzLTZV8tiDpdCNBypn78V4Dg==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
    arch "2.1.1"
    bluebird "3.5.0"
    cachedir "1.3.0"
    chalk "2.4.2"
    check-more-types "2.24.0"
    commander "2.15.1"
    common-tags "1.8.0"
    debug "3.2.6"
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
    fs-extra "5.0.0"
    getos "3.1.1"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    url "0.11.0"
    yauzl "2.10.0"

it works.

Previously, we used:

[email protected]:
  version "3.6.1"
  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.6.1.tgz#4420957923879f60b7a5146ccbf81841a149b653"
  integrity sha512-6n0oqENdz/oQ7EJ6IgESNb2M7Bo/70qX9jSJsAziJTC3kICfEMmJUlrAnP9bn+ut24MlXQST5nRXhUP5nRIx6A==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
    "@types/sizzle" "2.3.2"
    arch "2.1.1"
    bluebird "3.5.0"
    cachedir "1.3.0"
    chalk "2.4.2"
    check-more-types "2.24.0"
    commander "2.15.1"
    common-tags "1.8.0"
    debug "3.2.6"
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
    fs-extra "5.0.0"
    getos "3.1.1"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    untildify "3.0.3"
    url "0.11.0"
    yauzl "2.10.0"

note: that we're still using 3.6.1 in package.json. And we can confirm amplify is using 3.6.1 when running the test.

I tried this and now I’m not generating the aws-imports file anymore while I am doing the amplify simplePush step for the backend. Would be great if someone from aws could offer some assistance while I’m unable to deploy my app since October

[kahdojay]

@ascorbic the issue I'm referring to is not specific to PR previews but with large deploy artifacts. Do you have another GitHub issue I could look at that describes your problem? If not please feel free to open a new one so we can get some more context.

[kahdojay]

@kkesley-shine @rickiesmooth can you both please share your respective buildspecs (if possible, from the time you were experiencing the issue, or a buildspec with which you are able to reproduce the issue)?

Also @rickiesmooth if you could tell me a specific build number (for dmm4bu2tim2c; eu-central-1) that I can look at when this happens, it would help a lot. (We're going to start asking for these for new issues to save time)

[kahdojay]

@desertmonkey same for you if you are able, please - specific build number, as well as Amplify ARN (for the app id and region), and a buildspec with which you can reproduce the issue.

[kkesley-shine]

Hi @kahdojay

Here's our amplify.yml

version: 0.1
backend:
  phases:
    build:
      commands:
        - "# Execute Amplify CLI with the helper script"
        - amplifyPush --simple
    postBuild:
      commands:
        - nvm use $VERSION_NODE_12
        - yarn install
        - yarn run test:scripts
        - yarn run data-loader
        - yarn run test:integration
frontend:
  phases:
    preBuild:
      commands:
        - nvm use $VERSION_NODE_12
        - yarn install
        - yarn run test
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: build
    files:
      - "**/*"
  cache:
    paths:
      - node_modules/**/*
test:
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'
  phases:
    preTest:
      commands:
        - nvm use $VERSION_NODE_12
        - yarn install
    test:
      commands:
        - npm run ci:e2e
    postTest:
      commands:
        - npx mochawesome-merge --reportDir cypress/report/mochawesome-report > cypress/report/mochawesome.json

and scripts in our package.json

"scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test --testPathIgnorePatterns=\"src/integration\"",
    "test:scripts": "jest scripts/",
    "test:lambda": "jest --testPathPattern=\"amplify/backend/function\" --config=amplify/backend/function/jest.config.json",
    "test:integration": "react-scripts test src/integration",
    "test:all": "jest scripts/ && yarn test:lambda && react-scripts test",
    "ci:cypress": "cypress run --reporter mochawesome --reporter-options \"reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss\"",
    "ci:e2e": "start-server-and-test start 3000 ci:cypress",
    "eject": "react-scripts eject",
    "lint": "eslint . --ext .js,.jsx,.ts,.tsx",
    "data-loader": "node ./scripts/src/data-loader/main.js"
  },

Thanks!

And we also noticed that cypress 3.6.1 is getting stuck recently. And coincidently, there's a 3.7.0 out last week.

We also noticed amplify is installing cypress globally when provisioning the build. Do we have to use the one installed globally?

UPDATE

I tried installing 3.7.0 but it's still stuck in the testing phase.. Last log is:

Running: signup_spec.js (4 of 4)

It's master build #47 if you need the details..

But updating cypress to v3.7.0 in yarn.lock does not give me "Failed to deploy" error.

[wajjs]

I opened issue #287
"@wajjs how big is your node_modules folder? We have a known issue in #115 which we are actively working on. The temporary workaround is to delete unused branches in your app."

For me the deploy to hangs around 14000 files.
My node_modules folder is 350 MB.

[desertmonkey]

@desertmonkey same for you if you are able, please - specific build number, as well as Amplify ARN (for the app id and region), and a buildspec with which you can reproduce the issue.

Thanks @kahdojay, as my colleague @cawleyflower mentioned above, we changed our build procedures (including moving all our binary files to a CDN) and haven't seen the issue since.

For the record, our ARN is arn:aws:amplify:us-west-2:054500231008:apps/d1ev2ep2azk4pn

[kahdojay]

@kkesley-shine Thanks for sharing. For those having build issues with Cypress, please use version 3.4.1 as a workaround for now. We’ll update this thread once we’ve added full support to properly execute later versions. If builds continue to fail with 3.4.1, please let us know. @rickiesmooth

[ascorbic]

@kahdojay I'm getting failures during deploy with the same error, but not on the master branch. I'm not using Cypress, and the site is around 13k files. My ARN is arn:aws:amplify:eu-west-1:615693933970:apps/d301w3ojpkhzjt.

[kahdojay]

@ascorbic will take a look, can you also share the build number?
@wajjs will take a look, thanks for sharing your details in the other thread

[kahdojay]

@wajjs in #287 you provided a job arn that has a different account and app id than the app id provided. Can you confirm which app, branch, and job id we should look at?

[jordan314]

Hi, we are also having this issue, deploy stuck for three days, app d3mckmn7bfao41 branch master build 4 in oregon

[kahdojay]

@jordan314 there are a few different underlying issues being discussed in this thread. I should've asked for this earlier, but can you (and anyone requesting assistance) please provide more details, as provided in our updated issue template

This will help us save time in investigating and allow us to help more people. Thanks!

[jordan314]

Hi, we've provided more details in support ticket 6655323281, a lot of the content is sensitive

[ascorbic]

@kahdojay See build d301w3ojpkhzjt/search-page/7 for an example

[kahdojay]

@ascorbic This looks like a different issue, I see the build failed outright, you should see it in your console - if not please share a screenshot from that build of what you see. I also see an error regarding 'gatbsy-source-drupal' in the log that I believe you should as well.

[ascorbic]

@kahdojay No, the build succeeded. You must be looking at the wrong one. This is the one: https://eu-west-1.console.aws.amazon.com/amplify/home?region=eu-west-1#/d301w3ojpkhzjt/search-page/7

[kahdojay]

@ascorbic thanks, you're right I was looking at the wrong one. It looks like you're running into the large artifact issue as well. We are actively working on a fix for this, in the meantime if you have any unused branches, you can try deleting them. Another potential workaround would be to deploy to a separate Amplify app.

[ajhool]

@kahdojay

It seems like there are multiple issues going on in this thread and the request to "provide more details" is difficult when the only details provided from the Console logs are:

2019-12-28T21:31:00 [INFO]: Starting Deployment
2019-12-28T21:31:00 [ERROR]: Failed to deploy
2019-12-28T21:31:00 [INFO]: Deployment finished.

We're deploying a basic Vue.js app (the app is generated using the official vue-cli provided by the Vue.js team). We are using one branch although would like to use many branches for gitflow as advertised by Amplify Console (https://dev.to/kkemple/branch-based-deployment-strategies-with-aws-amplify-console-1n3c)

Our app id is: d1v4fadcbu6950

Desired functionality:

• The app deployment should work correctly or basic debugging output should be provided
• A troubleshooting page should be available for known issues (eg. deployment size, cypress cache, etc.)
• Amplify Console should be proactive in providing error feedback, rather than coming to github to post an AppId to find out which known-but-not-documented issue we are running into.
• Cypress should not interfere with deployment, as advertised by Amplify Console, here: https://docs.aws.amazon.com/amplify/latest/userguide/running-tests.html

[hellokingdom]

arn:aws:amplify:eu-west-2:673653084816:apps/d26tysq5iwv37v

Failing to deploy if tests are enabled.

The tests complete successfully but instant fail on deploy step.

[germain-receeve]

Same issue here, downgrading to cypress 3.4.1 didn't help.
Did anybody figured out a way around this issue?

[swaminator]

@ajhool @germain-receeve @hellokingdom we are aware of the Cypress issue causing deploy failures. Please refer to #283. We are actively working on a fix here and will let you know when that is rolled out. Apologies for the inconvenience. In the meantime to unblock yourself, set an environment variable called USER_DISABLE_TESTS to true to disable your tests from running.

[cslogan-red]

Quick update: fixes are pending release. This issue has two parts (and each applied only when using some type of test suite via the test phase); an improperly bubbled exception scenario resulting in builds that either; hung until the build timeout, or builds that showed "successful" in the Console and immediately failed to deploy, each without logging the full exception details, as well as a command update.

For existing apps, the latest version of mochawesome-merge required a slight change to the final postTest command. Below is the correct test section edit for Cypress 3.4.1^ & mochawesome-merge 3^ (our docs are being updated to reflect this and you can apply this now to get tests working with these versions):

Final postTest command should be updated to:

_npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json_

Additionally, the first preTest command has been changed from "_npm install_" to "_npm ci_" to ensure the Cypress binary is always downloaded as this seems to be more reliable. These changes will get Cypress correctly running again, using their kitchen-sink repo as an example.

As stated above, these fixes are pending release and should be GA by 1/23. These updates will cause builds with actual exceptions that occur during the test step to be correctly logged to the Console's Test log, and the Test step will fail as desgined, as opposed to the previous behavior observed.

[spencer-aerian]

@kahdojay No, the build succeeded. You must be looking at the wrong one. This is the one: https://eu-west-1.console.aws.amazon.com/amplify/home?region=eu-west-1#/d301w3ojpkhzjt/search-page/7

Taking the mantel from @ascorbic as he's moved over to other things , we are STILL seeing this Failed to Deploy error in the Deploy console. Whats' the actual ETA of this getting into production? Tests disabled, verify is disabled etc.

[z3by]

We are facing the same issue here, is this gonna be solved soon?

[eterry1388]

Same issue for us. Build works fine without specifying test. But as soon as we put in test in the amplify.yml config in any fashion, it always fails.

[MrBokeh]

I found a working solution by running custom script such as node cypress.js and then merge the report with mochawesome-merge... which is quite hacky...

test:
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'
  phases:
    preTest:
      commands:
        - npm install
        - npm install wait-on
        - 'npm start & npx wait-on http://localhost:3000'
    test:
      commands:
        - npm run cypress:custom
    postTest:
      commands:
        - npx mochawesome-merge --reportDir mochawesome-report > cypress/report/mochawesome.json

[AndriiChubariev]

Hi,
I have started to experience the same issue today. The build is green, failed to deploy.

Using Gatsby site with a lot of images. Used to work fine on Saturday.

[sashasochka]

We are having the same issue.

[z3by]

Please re-open this Issue, has it been solved?

[AndriiChubariev]

For more details, my build artifact is about ~1Gb, a lot of images. I had two environments: dev and prod.
After I removed dev and left only one environment the issue just fixed itself and not reproducing anymore.

I understand that it is not an option for many users to delete a dev environment. It worked in my case just because it is a hobby project.

[deybhayden]

Hey there - are there any updates on this issue? We have a build artifact that's 55 MB but is still suffering the same fate. Is there a recommended artifact size to avoid this problem?

[cslogan-red]

This issue has become a placeholder for three distinct issues that all caused similar behavior in Amplify Console, a Deploy step with a sometimes immediate (and sometimes after deploying many files) "Failed to deploy" message.

The first issue has been fixed and released as described above and was related to the use of the Test step, exceptions here weren't always being caught and bubbled up to your build logs properly, so the Test step was actually failing and it was not being bubbled up, resulting in the "failed to deploy" behavior. Note this doesn't mean you'll never have failed test steps, this step can fail for dozens and dozens of reasons, you're just going to see it as a test failure with proper logs now

The remaining two issues are being actively worked on, the highest priority of which has a release coming soon, and we'll update this thread once the release is available in all regions.

For these remaining two issues, the failure to deploy is compounded by two things, the number of branches you have connected to a single Amplify Console app, and the number of deployed files you have in your app.

We're seeing this issue most commonly in (very) large static Gatsby sites with many static images; the suggested workaround is the same no matter what, reduce the number of branches you have connected to each app down to a single Production branch if necessary (you can connect multiple apps using the same repo using a single branch per app if you have to, in order to still test each branch). The forthcoming update will resolve this limitation.

[deybhayden]

Hey there - thanks for the update @cslogan-red! I wanted to drop a note that the interim fix did work for our project - deleting everything but the single branch. This _included_ the PR preview branches as well.

[gianlucapisati]

Hi I have the same error, even if deleting all the other branch except the production one...
this is my arn arn:aws:amplify:eu-west-1:135446448955:apps/d1xjyanq2k93xx

[FemkeBuijs]

I also still experience the same problem.
With cypress tests the error 020-03-19T10:45:32 [ERROR]: Failed to deploy remains, while the tests are successful. Build settings:

 test:
   artifacts:
     baseDirectory: cypress
     files:
       - '**/*.png'
   phases:
     test:
       commands:
         - 'npx cypress run'

When commenting out these settings there is no error and the deploy is successful.

[jordan314]

Honestly our solution was to stop using amplify and use codebuild / S3 / cloudfront. Something like this
https://medium.com/@muaohua/deploy-your-single-page-application-to-s3-and-cloudfront-with-code-build-and-code-pipeline-4ae6e5c12018

[robmarshall]

I am still getting this issue

[amuresia]

+1

[grudra7714]

+1

[plumbis]

Is there an ETA for a fix for this issue? The current workaround of deleting the app and recreating it is a major challenge since it requires updating DNS, environmental variables and more. This has impacted our production environment multiple times now.

[litwicki]

@plumbis the team is actively working on this as I type and planning to have an update in the next several days. I or the team will update this thread as soon as we have a more information to share.

[robmarshall]

I needed to deploy my clients projects sooner rather than later so cracked on with Codepipeline. This walks through the steps needed: https://thoughtsandstuff.com/deploy-gatsby-aws-with-ci-without-amplify

[adriencg]

We're currently working on testing & deploying the fix, and sincerely apologize for all the inconvenience this has caused.

[stigalvestad]

Is there any news on this case, @adriencg ?

[michalmikolajczyk]

Hi we have the same problem. Our app ARN arn:aws:amplify:us-west-2:329268793000:apps/d2o4y68d25axi5

This seems related to the tests. The problem only started showing after I added the test: paragraph to amplify.yml and when it was removed, the build deploys. With that section, I get

2020-05-05T08:48:40 [INFO]: Starting Deployment
2020-05-05T08:48:40 [ERROR]: Failed to deploy
2020-05-05T08:48:40 [INFO]: Deployment finished.

I worked around this problem by moving the commands for tests into the _preBuild_ phase for frontend (I just added one command yarn test tbh)

[d-luk]

Thanks for the workaround @michalmikolajczyk.

We are having the exact same problem. With the test step the deployment fails, but without it it succeeds. We are also just running yarn test on our Create React App project.

@adriencg Can you give an update on the fix you mentioned?

[steinybot]

Are the remaining issue related to the size of the deployment artifacts size? As far as I know my artifacts are not large.

My cache is probably very large though. Is there even a way to invalidate/expire the cache? The docs are very light on the subject.

I respect that fixing this issue may be complicated but this is a "P0 - loss of major functionality - do not pass Go do not collect $200" kind of bug and it has been quite a long time. Can we at least get a more detailed error message so we have some hope of working around it ourselves?

I really want to like amplify but I have encountered so many different bugs and usability issues in the past 2 weeks of using it that I'm not sure it is worth the hassle. Most of the time I have just resorted to deleting everything and starting again which sadly works but my good will is running low.

[steinybot]

I noticed something odd with my builds.

The first one which failed with the infamous "Failed to deploy" message actually has the Test phase marked as pending. I'm not sure if that has been mentioned before or not.

The next one where I tried removing parts of the test phase shows the opposite problem. It shows the build phase as having failed but it was actually the test phase (I had not specified a baseDirectory for the artifacts).

In case anyone wants to take a closer look the App ARN is arn:aws:amplify:ap-southeast-2:844681392632:apps/d1855l7ntw7ham, preview build pr-5, builds 3 and 4.

[simpson]

We've been experiencing this over the past few weeks

[simpson]

Before I go and delete all of our PRs, can someone confirm if that actually fixes anything?

[itdoh]

We've been experiencing this over the past few weeks too.
#651

[robmarshall]

Any updates?

[Ensive]

Experiencing this issue as well. Will try to readd branch or switch off the tests 🤦🏻‍♀️

[ryanwilsonperkin]

We got an update from our support rep today that a fix is in progress and ETA is 8 weeks

[Gowiem]

Just adding my experience to the chorus here so I can follow along. Experiencing this issue for a client unfortunately.

8 weeks seems like a pretty long time considering multiple Amazon team members have jumped on the thread and said a release was imminent. Agreed to whoever called this a P0 bug... Seems like a deal breaker if you have to tell service users to delete a branch / part of their setup.

[simpson]

Unfortunately the frequent problems with Amplify have forced us to look for other services that are more stable and production ready. It will be an amazing product, they just need to invest in getting the capabilities to a steady state

[larryle]

I got same issue, any update pls...

[larryle]

Just made some progress:
1: In package.json, I changed
"build": "ng build" to
"build": "ng build --outputPath=dist/",
Now the amplify publish command finished successfully.

2: But now I got a new problems, when I tried open the APP in web browser. I can only see a blank screen, nothing showed up

Here are more details
1) I am trying to make simple app to follow this TUTORIAL,
https://docs.amplify.aws/start/getting-started/hosting/q/integration/ionic#publish-your-app

2) I am using latest Ionic 5, ionic info:
Ionic:

Ionic CLI : 6.10.1 (/usr/local/lib/node_modules/@ionic/cli)
Ionic Framework : @ionic/angular 5.2.2
@angular-devkit/build-angular : 0.901.9
@angular-devkit/schematics : 9.1.9
@angular/cli : 9.1.9
@ionic/angular-toolkit : 2.2.0

Cordova:

Cordova CLI : 9.0.0 ([email protected])
Cordova Platforms : ios 5.1.1
Cordova Plugins : cordova-plugin-ionic-keyboard 2.2.0, cordova-plugin-ionic-webview 4.2.1, (and 4 other plugins)

Utility:

cordova-res : 0.15.0
native-run : 1.0.0

System:

ios-deploy : 1.9.4
ios-sim : 8.0.2
NodeJS : v14.4.0 (/usr/local/bin/node)
npm : 6.14.5
OS : macOS Catalina
Xcode : Xcode 11.5 Build version 11E608c

Maybe not compatible with IONIC 5?

[litwicki]

@larryle @desertmonkey @simpson @Gowiem @ryanwilsonperkin @robmarshall forgive me for not following up on this issue. I dropped the ball and that's on me. I do want to assure everyone that this is our highest operational priority right now.

When I posted several months ago we did believe we had a fix nearly ready but opted not to roll that out as it was frankly not a comprehensive fix that addresses the underlying problem. Unfortunately that problem has required several months of work, but is in progress. This is something I am tracking daily and have a focused team on specifically to address comprehensively such that there will be no scenarios in which deployments fail when we are complete.

As of this post we are targeting mid-August as the date we will beginning rolling this out to all regions. We are making sure to approach this differently and comprehensively this time. Anyone who wants to ping me directly can feel free to do so and I'd be happy to discuss any additional details.

Lastly I just wanted to reiterate that @swaminator and I are treating this as our single highest priority and we will make regular updates to this issue when we have anything to share.

[jairenee]

I have a new app, only one branch. Artifact size is 200MB. This is a ridiculous bug to exist for so long, and I think we'd all feel a lot better if the issue could be explained and addressed somewhere. The fact that we're waiting for a fix with no explanation whatsoever is a huge slap in the face.

[litwicki]

@jairenee the underlying issue is not precisely the size of the artifact, but moreso the total _count_ of files (including the sum of all branches). If you'd like us to look specifically at your app I'd be happy to connect you with the team and we can see about a possible workaround while the rest of the team continues to work on the fix.

[jairenee]

@litwicki So if I'm understanding the issue has nothing to do with the artifact at all then. That's interesting. I'll consolidate to just one branch for development while this is being worked on. I've reached out to you on Chime.

[litwicki]

Just wanted to share an update that we are still actively working on this and on track.

[Galaf]

@litwicki This issue started to appear from today onwards for me.
Weirdly enough, I don't have a lot of branches connected right now and I don't think the count of files in one single deployment might be the issue since I've been working on fixes rather than addition of new features.

Although one thing that I can think of right now is that I merged a lot of branches today.
Does it then mean there is some up-limit to the number of build jobs and deployments that can happen in a span of time?

Could you give more information as to what is happening right now and if there is some way to remediate it on my side of things, if any?

[litwicki]

@Galaf Unfortunately it will not matter how few files you change per branch, but in the number of branches. Each branch will effectively be additive in the number of files, with each branch being a clone with respect to number of files. There is no limit to the number of build jobs nor deployments. The current workaround would be to limit the total number of branches.

[Galaf]

@litwicki I encountered the issue with four branches (develop and master branches included).
Is there a determined number of branches that are supported?

Will that number be increased with your incoming fixes?
If so, can we get a time frame regarding releases of these?

Thank you for your time.

[litwicki]

@Galaf There is no explicit limit to the number of branches supported; currently this is merely because of the issue we are actively working on resolving. The current issue again is the number of aggregate files in all branches. You may have 1 branch with 50,000 files, or 5 branches with 10,000 files, and both could encounter this issue (there is no definitive number I can share as it varies by app, but it is in the tens of thousands).

The work we are doing now will eliminate this problem altogether, such that there will be no limit to the number of files nor the number of branches you can use with your app. Currently we are targeting end of July to begin testing this, and pending the outcome of testing I will be sharing frequent updates on when we hope to have this resolved for customers.

I've asked @cslogan-red from the team to add some additional info here as well.

[cslogan-red]

As @litwicki mentioned, the root problem comes down to the aggregate file count across all branches. The team's original deployment and hosting designs required this component, and the file contents of every additional branch you connect counts toward this aggregate file count limit. Additionally, the popular PR Previews feature compounds this, as internally, this component treats open PR's the same as branches, thus contributing to the overall aggregate limit. We've found folks may only have two or three branches connected to the Console, but may have dozens of open PR's connected as well, and this compounds and eventually results in the same problem. Merging and closing open PR's connected to the Console has the same benefit as disconnecting non-production branches in that it decreases the total aggregate file count.

As a result, we took a step back, and I've largely redesigned our hosting and deployment modules. We needed to be sure that we could deliver something that would unblock every customer (there will be no aggregate file limit) while also not impacting any of our existing customer's hosted apps (the update will eventually just be live for all apps, with zero downtime or impact). We believe we have that solution, and as Jake mentioned, the team is hard at work on implementation and soon testing of the updated designs.

Again, as Jake has already said, getting this right for you, our customers, is presently our highest priority and we've worked hard to get to a solution that will work for all of our customers, and we're still currently hard at work on the implementation. We will continue to share updates in this thread as testing progresses.

[rgevrek]

Hi

I got also following error on deployment after a successful test phase with angular:

2020-07-02T09:10:04 [INFO]: Starting Deployment
2020-07-02T09:10:04 [ERROR]: Failed to deploy
2020-07-02T09:10:04 [INFO]: Deployment finished.

The dist folder is with 432 KB very lightweight and I have a single branch in CodeCommit (master).
But if I skip test phase, the deployment works.
And I've tried also a sample with Vue, that also works.

[abhi7cr]

Hi @rgevrek,

Can you share your appId and region?

[rgevrek]

Hi @rgevrek,

Can you share your appId and region?

Hi @abhi7cr

the App ARN is arn:aws:amplify:eu-central-1:965927982908:apps/d3ihuykhg95t7t

[litwicki]

I wanted to share a progress update on where we're at with this as a team. What we are starting to test this week is a resolution to two of the primary scenarios triggering this failure. We are hoping to begin previewing this fix with customers in the coming 1-2 weeks. If you are interested please let me know and we can coordinate.

The next major milestone to completely eliminate this issue will be addressing the hosting component as @cslogan-red pointed out above. We are beginning that work now in parallel and will be previewing with customers when we can for that as soon as possible. Currently hoping to do that in mid to late August.

As always if you have questions or comments please let us know.

[ganipcanot]

@rgevrek Can you please try adding an artifacts section under the test section in your buildspec similar to here: https://docs.aws.amazon.com/amplify/latest/userguide/build-settings.html

be sure to include a baseDirectory as well (It can be anything to upload, just so long as its a valid path).

test:
  phases:
    preTest:
      commands:
        - *enter command*
    test:
      commands:
        - *enter command*
    postTest:
      commands:
        - *enter command*
  artifacts:
    files:
        - location
        - location
    configFilePath: *location*
    baseDirectory: *location*

Here's a cypress example: https://docs.aws.amazon.com/amplify/latest/userguide/running-tests.html for reference.

[dnorak]

@litwicki I'm also encountering this issue since I add Cypress test last week.
It would be great if you could figure this out mid August.
I'm interesting to know when i'll could try to deploy again.

There is my app ARN : arn:aws:amplify:eu-west-1:180377447306:apps/d255ctmrcgd22k

Thanks

[litwicki]

@dnorak that may not be related actually. If you take a look at what @ganipcanot shared above, unless I'm mistaken that should help. If not, please let us know.

[rgevrek]

Hi @ganipcanot

I have to add configFilePath: '**/mochawesome.json' in amplify.yml under configFilePath.artifacts, but therefore I replaced (as here is an example) protractor from Angular CLI with codeceptjs for using mochawesome reports (npx codeceptjs run --reporter mochawesome).

Now the e2e tests are working and got displayed in the test results, but are there also plans to enable other report formats other than mochawesome? Unit tests in angular are using karma and there is no adapter for mochawesome to work with karma, but a lot of other report formats as mocha, jasmine specs, istanbul etc.

[litwicki]

I wanted to share an update on our progress as we're testing this week. We are still very interested in working with any customers who would like to help us preview these changes, and are being very deliberate with how we test and eventually roll these out later this month.

[mattpocock]

customers who would like to help us preview these changes

Yes please - let me know what info you need.

[litwicki]

@mattpocock please contact [email protected] and reference this issue and we can coordinate from there, thanks!

[fkoner]

@litwicki We have the same situation here. Any solution we can carry out?

[gcphost]

Ran into this issue today. I retried the deployment and it worked.

[fabioalencar]

Same issue here. App ARN: arn:aws:amplify:us-east-1:598366504733:apps/d16uqynew5iqbx
Branch "develop" fails on deploy step. Retried the deployment several times with no results.

[marko-asplund]

Any ETA for making a fix generally available?

[litwicki]

@marko-asplund We're working with customers this week to preview our changes, and will update end of week on next steps.

[marko-asplund]

@litwicki We'd like to participate in previewing, if possible. I sent an email about this to [email protected] on wednesday.

[marko-asplund]

@litwicki Any news on next steps?

[litwicki]

@marko-asplund sorry we didn't respond again last week, we've been processing through all stages with our testing. However, we're planning to proceed this week with deploying to all regions. We will update this thread when we have more concrete detail.

[marko-asplund]

@litwicki Any news on this? 😃

[litwicki]

@marko-asplund we're beginning to roll out to all regions this week, under extreme caution. As new regions are available we will update this thread. Thanks for checking in and keeping me honest!

[vankessel-it]

@litwicki That sounds great, I hope it works, we've just added a new language to our website and it won't deploy anymore.

[fitzgers]

@litwicki Please do keep us updated... Appreciate all the help in wrapping it up. I'm extra curious about us-west-2, as that is where my current headache around this resides....

[vankessel-it]

@litwicki eu-central-1 is not working yet. Tried it 8 hours ago and just now. I'll try again tomorrow. It is surprising, in the end it is just one branche with a 14000 tiny files, most of which remain unchanged. A few months ago we worked around it by moving all images to a bucket... but we have no such option now. Hope you can fix it, cheers, Marc

[litwicki]

@fitzgers @vankessel-it We will update as new regions are live. As of today we are live in eu-west-2 and us-east-2 and in progress with ap-southeast-1

As new regions are live we will update this thread.

[tscole]

@litwicki just been testing a copy of one of my projects in eu-west-2. Was previously failing (in eu-west-1) in the deploy stage. It's a static site with around 32,000 files output by the build stage. Has successfully deployed in eu-west-2 and the whole process took less than 6 minutes. Previous attempts to build were taking 30+ mins and failing.

So this looks pretty promising! Thanks. Looking forward to seeing it roll out in eu-west-1

[marko-asplund]

Sounds good! 👍 Any ETA for eu-west-1?

[litwicki]

@tscole that's great to hear, please let us know if you have any issues or recommendations for additional improvements!

[frazbot]

@litwicki our builds have started hanging at the deploy phase today on us-east-2. Could this due to the update?

[cslogan-red]

@frazbot can you please provide your app Id so we can take a deeper look at logs?

[Ustrave249]

@cslogan-red we're having an issue with arn:aws:amplify:us-east-2:626294045888:apps/dxfl9a7zsfcnt

[cslogan-red]

@frazbot and @Ustrave249 can you each cancel any pending build and redeploy? We've made a small tweak on our end that we believe to be the culprit of the behavior you were seeing.

[Ustrave249]

Thanks @cslogan-red that's fixed it

[richardtapendium]

Is anyone else having issues now with branches not being picked up? I'm unable to enable pull request previews, it asks me to authenticate with github, which I do, and then I get presented with the same screen:

[bwalsh]

Build works

2020-09-21T18:56:10.298Z [INFO]: info Done building in 135.168 sec
2020-09-21T18:56:11.667Z [INFO]: Done in 136.59s.
2020-09-21T18:56:11.672Z [INFO]: # Completed phase: build
2020-09-21T18:56:11.674Z [INFO]: ## Build completed successfully
2020-09-21T18:56:11.769Z [INFO]: # Starting caching...
2020-09-21T18:56:11.856Z [INFO]: Creating cache artifact...
2020-09-21T18:56:35.102Z [INFO]: # Cache artifact is: 734MB
2020-09-21T18:56:35.183Z [INFO]: # Uploading cache artifact...
2020-09-21T18:56:42.967Z [INFO]: # Caching completed
2020-09-21T18:56:43.071Z [INFO]: # Starting build artifact upload process...
2020-09-21T18:56:57.274Z [INFO]: # Build artifact is: 93MB
2020-09-21T18:56:57.292Z [INFO]: # Build artifact is: 93MB
2020-09-21T18:56:57.396Z [INFO]: # Uploading build artifact '__artifactsHash.zip'...
2020-09-21T18:56:57.398Z [INFO]: # Uploading build artifact '__artifacts.zip'...
2020-09-21T18:56:58.673Z [INFO]: # Build artifact upload completed
2020-09-21T18:56:58.783Z [INFO]: # Starting environment caching...
2020-09-21T18:56:58.784Z [INFO]: # Environment caching completed

But deploy fails

2020-09-21T18:57:06 [INFO]: Starting Deployment
...
2020-09-21T19:03:57 [ERROR]: Failed to deploy
2020-09-21T19:03:57 [INFO]: Deployment finished.

One branch works, the updated master fails.

[iaguirre88]

I'm facing a similar issue. In my case, the test phase succeded but then it gets stuck in the deploy phase without raising any error.

2020-09-21T20:38:00 [INFO]: Starting Deployment
2020-09-21T20:38:00 [INFO]: Updating Edge config
2020-09-21T20:38:01 [INFO]: Start deploying test artifact
2020-09-21T20:38:01 [INFO]: Deploying videos/***.mp4
2020-09-21T20:38:01 [INFO]: Deploying videos/**.mp4
2020-09-21T20:38:02 [INFO]: Finish deploying test artifact

[jorgekleeen]

Same thing as @iaguirre88 here.

This is happening only in one of our configured environments. Build takes more than the _BUILD_TIMEOUT environment variable and never fails.

Even we cancel, and re-deploy, the output is the same.

AppId: d3ig44i44gjc6s

Build No: 84

Branch: staging

Output:

...
2020-09-21T20:43:30 [INFO]: Deploying report/mochawesome-report/mochawesome.html
2020-09-21T20:43:30 [INFO]: Deploying report/mochawesome-report/mochawesome.json
2020-09-21T20:43:30 [INFO]: Finish deploying test artifact

Region: us-east-1

Summary:

[litwicki]

@jorgekleeen @iaguirre88 @bwalsh we are looking into this now.

[kahdojay]

@jorgekleeen @iaguirre88 @bwalsh can you add appId, region, branch, and build number details?

[bwalsh]

us-west-2
my work around was to delete down to only 1 branch

[iaguirre88]

@kahdojay Is there a way to send this information privately?

[kahdojay]

@iaguirre88 yes you can email us at [email protected]

[iaguirre88]

@kahdojay Any update on this? Thanks

BTW, I just realised that skipping the test with USER_DISABLE_TESTS makes the deploy succeed but it's weird since the test were all passing

[bwalsh]

The problem is now happening even with just one repository (previous workaround).
Now in order to deploy, I need to reduce functionality by reducing number of files.

Build artifact is: 93MB
8419 static files

On Tue, Sep 22, 2020 at 6:05 AM Ignacio Aguirrezabal <
[email protected]> wrote:

@kahdojay https://github.com/kahdojay Any update on this? Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws-amplify/amplify-console/issues/115#issuecomment-696707904,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAALVQCTZVKPYL2GBDIWAELSHCOLHANCNFSM4IWULLIQ
.

[kahdojay]

@jorgekleeen @iaguirre88 we've identified a fix and will start rolling it out today.

[kahdojay]

@bwalsh it looks like your failure is related to a different reason that we'll need to keep investigating

[kahdojay]

@jorgekleeen @iaguirre88 can you let us know if you're still experiencing the issue in us-east-1 (and if so let me know the build number / branch name)

[bwalsh]

@bwalsh it looks like your failure is related to a different reason that we'll need to keep investigating

@kahdojay thanks for following up. much appreciated

[jorgekleeen]

@jorgekleeen @iaguirre88 can you let us know if you're still experiencing the issue in us-east-1 (and if so let me know the build number / branch name)

@kahdojay just ran a couple of consecutive builds and the build is now being deployed as expected! Thanks a lot! 👏

[iaguirre88]

@kahdojay It seems it's working now 🎉
Thanks for working on this 🙌

[kahdojay]

great to hear it! will get the fix out to all regions today, and investigate the unrelated deploy failures separately

[davetlewis-van]

I had builds that were working perfectly in us-east-1 yesterday and earlier today with the issue resolved:

2020-09-25T16:32:44 [INFO]: Beginning deployment for application d3idu9b8upbkgj, branch:master, buildId 0000000026
2020-09-25T16:32:47 [INFO]: Got archive: 271284892 bytes
2020-09-25T16:32:49 [INFO]: Archive unzipped: 21844 files
2020-09-25T16:34:55 [INFO]: Deployment complete

But now it seems to have reverted to deploying individual files:

2020-09-25T21:17:46 [INFO]: Starting Deployment
2020-09-25T21:17:46 [INFO]: Uploading ...
2020-09-25T21:17:46 [INFO]: Skipping upload of existing file ...

Did the fix get rolled back in this region, or did I jump the gun and it been fully tested and deployed yet?

Thanks!

[MrBokeh]

I have been following this thread for a year now. I got the pipeline working perfectly for a client project since November 2019 and my own project since August 2020 but I want to share a few comments with the sample amplify.yml template:

1. For js, ts, Node.js apps... we need to change the node version to 12 on every single step... otherwise, it will fail without error logs, the default Node.js version is pointed to 10 which cause a lot of issues
2. If the amplify YAML file is not valid on your git repo and you try to connect it to a new amplify app, the new app will get created but you won't be able to connect it to the repo
3. If I try to call a webhook in the YAML file, it will be considered as an invalid template but it will work only through the git repo continuous way (not manual) after the successful creation of the amplify app.
4. The wait-on step is not working anymore with sample Amplify template for any PWA... you can use anything else other than wait-on
5. Mocha report step is not working since 2019, since the mocha report can't create files on a specific folder or path, you need to write your own steps and call it on the pipeline...
6. If the pipeline fails try to remove a step to see which step is not working...
7. Cypress need to point to a specific version otherwise it will fail due to the incorrect version between your app and the one on the VM

So, write your own YAML file and avoid the template... at least for now since 12 months ago...

[fitzgers]

@litwicki I'm guessing the update has been rolled out to us-west-2, and I just want to note here I've seen a large improvement starting at some point on Thursday Sep 24. Our build times (specifically the Deploy step) have reduced considerably, and we've seen no failed deployments in the last 2 days. Thanks!!

[marko-asplund]

great to hear it! will get the fix out to all regions today, and investigate the unrelated deploy failures separately

@kahdojay I'd just like to double check that a fix for the original issue of deployment failures has been deployed to eu-west-1?

[bwalsh]

@bwalsh it looks like your failure is related to a different reason that we'll need to keep investigating

@kahdojay any update?

[litwicki]

@bwalsh we're still investigating and testing a fix, are you still unable to build/deploy your app?

@fitzgers really excited to hear that. We're still rolling out all these changes and hoping for improvements to continue!

@marko-asplund We're pausing over the weekend as we saw some issues overnight; are you experiencing issues still?

@davetlewis-van We did rollback in some regions while we resolve an issue we identified. All builds should be working so please let me know if you experience any issues.

@MrBokeh I'd love to hear more about how we can improve. Thanks so much for your feedback.

[bwalsh]

@bwalsh we're still investigating and testing a fix, are you still unable to build/deploy your app?
I needed to reduce the number of pages in the deploy in order to get it to work.

[lukehedger]

Still seeing intermittent deployment failures in eu-west-2:

2020-09-28T13:17:10 [INFO]: Updating Edge config
2020-09-28T13:17:10 [ERROR]: Failed to deploy

App ID: d9slrnc36kbq7

[davetlewis-van]

@litwicki Hi Jake, Yes, unfortunately, I'm still running into issues. Builds on both my staging and master branch are failing on the Deploy step.

[litwicki]

@davetlewis-van we're working on rolling out changes today to resolve any issues. I will have the team follow up as soon as we have an update.

[fitzgers]

@litwicki Something interesting: We're not seeing the deploy failures any more, but I have noticed that the improvement in our build/deploy times mentioned here have regressed back to what they were originally. We were down to around 6 minutes on Friday of last week, and have creeped back up to 12 or so today. No significant code or content changes on our end. This is us-west-2, btw.

[litwicki]

@fitzgers This is likely due to our fallback while we roll out fixes today. If this continues through next week please let me know but I would expect it to resolve end of week if not sooner.

[yakkomajuri]

We're supposedly on the "Beta" but can't get a build to work if we have >8 PRs open. From today:

[GooBall]

I'm still getting failed deploys in eu-west-1.

[litwicki]

@yakkomajuri @GooBall We are planning to restore the new changes as early as today. We will update this thread when we have begun rolling out.

[AustinZhu]

Still get stuck in the deploy phase with 300MB+ build size in ap-northeast-1. Takes too long to upload dependency files.
Only one branch and no preview.
arn:aws:amplify:ap-northeast-1:564383102056:apps/d2dh1n7ljztg6f

[fitzgers]

@litwicki Any update? We have noticed our build times (which had improved greatly) have regressed back to what they were previously, and we are back to having failed deployments in us-west-2.

[litwicki]

@fitzgers @AustinZhu we are re-enabling individual regions after they have been verified with the appropriate updates. That will begin today with eu-west-2.

[tscole]

After previous successful deployments in eu-west-2 I am now seeing issues again. As before, large static side with tens of thousands of files is hanging in the deploy phase.

App ID: d3iczrv56cp0p5

[litwicki]

@tscole we've re-enabled the new deployer service in eu-west-2. We're not seeing any failures, but can you please let us know if you're not able to proceed?

[tscole]

@litwicki it seems to be working today. I can see both my environments appear to have successfully deployed. Will let you know if I see any further problems

[isi-gach]

@litwicki, do we know when would be the rolling out to eu-west-1?

[hsspain]

The new deployer service has been enabled in us-east-2. We will update this thread as more regions are enabled.

[litwicki]

@isi-gach I can't give an exact date yet, but I can say it's in progress.

[petetak]

In the past few days we started getting this issue in us-east-1 with the deployment error
2020-10-14T08:42:03 [ERROR]: Failed to deploy
It's causing huge problems for my client so if there are any workarounds we could use in the meantime then I'd really appreciate it!

[tomlandau]

@litwicki
hi, we're experiencing the same issue, for app: arn:aws:amplify:us-east-1:161809446696:apps/d16bd6505exenn.
Can you please take a look?
We have only one branch, and no tests configured.

[petetak]

We have a few branches on the repo but not a lot more than we've had for a while. We don't have tests configured. The app is arn:aws:amplify:us-east-1:846087266623:apps/d3icy4rih41kkd

[litwicki]

@petetak @tomlandau we are looking into this now

[fitzgers]

@litwicki To add one to your list, we are now experiencing a deployment failure on pretty much every push. This seems to have regressed in the last couple of days. I can usually fix by re-deploying, but it is a laborious process to say the least. We're down to one branch and no configured tests.... Our app is arn:aws:amplify:us-west-2:790675203832:apps/dhc11nf9dhtm2

[petetak]

@litwicki would you mind also mentioning @jbradday in your response as I'm not in the US timezone - thanks!

[litwicki]

@fitzgers Sorry to hear about the trouble you're having. We are rolling out a change tomorrow morning that will be resolving this for us-east-1 and will follow up with you directly to verify when it's live.

[fitzgers]

@litwicki Thanks! We're in us-west-2, so any update on status for that region would be great!

[halimb]

@litwicki I'm facing a similar issue here on eu-west-1 however, it happens when I have cypress test artifacts. Any idea how to solve this? (while still uploading the files)

[litwicki]

@halimb We're hoping to be live in eu-west-1 tomorrow 10/16; I will follow up with the team here. @fitzgers us-west-2 is in progress now and likewise we will update here as soon as it's live.

[GooBall]

@litwicki Thanks for the update. We were having issues when too many PRs were connected and clearing them down would fix the problem. However now we have some branches that simply do not deploy at all and nothing seems to be fixing them in eu-west-1.

[gfest]

Hi @litwicki, any news on this? I'm having the same problem for a project in eu-central-1 since this morning: arn:aws:amplify:eu-central-1:415944032879:apps/d13fes5e3vrcjo

[litwicki]

@GooBall we're still not quite done with eu-west-1 but on track for this afternoon Seattle time.
@gfest We are in progress with eu-central-1 early next week, as with previous updates, as soon as we're live there we will update this thread.

[litwicki]

@GooBall took longer than we had originally planned but we're officially live in eu-west-1 now as well. Please let us know if you see any issues!

[acsrujan]

Hey, is the fix rolled out to ap-south-1? Is there a place where we can track which regions is the fix rolled out to?
Facing an issue with few branches, despite size being less than 100mb. Failing at Uploading to Edge.

[sugi-s]

I'm facing similar issue where the deployment hangs when adding test phase using testcafe framework....moving the the tests to build phase works....prior to the fix over the weekend the deployment phase would fail "Failed to deploy" with no meaningful error message but now the deployment phase hangs

[sugi-s]

@litwicki to add a little more context:
we are running amplify builds in eu-west-1
After the fix the deployment stage never completes with the following message in the console :

 [INFO]: Starting Deployment
 [INFO]: Updating Edge config
 [INFO]: Start deploying test artifact
 [INFO]: Deploying report.json
 [INFO]: Finish deploying test artifact

any help would be appreciated..... As a workaround moving e2e test to pre build phase seems to work but unable to get the test artefacts when running the e2e tests as part of pre build phase. Is there a way to get the test artefacts from build phase ?

[GooBall]

@GooBall took longer than we had originally planned but we're officially live in eu-west-1 now as well. Please let us know if you see any issues!

Thanks @litwicki, everything seems to have run ok today! I'll let you know if that changes.

[litwicki]

@sugi-s Are you seeing this issue since 10/16? I'll have the team investigate either way but please let us know.

[litwicki]

Hey, is the fix rolled out to ap-south-1? Is there a place where we can track which regions is the fix rolled out to?
Facing an issue with few branches, despite size being less than 100mb. Failing at Uploading to Edge.

@acsrujan We haven't published a timeline because it is very indeterministic unfortunately. As soon as a new region is live we're updating this thread. We are hoping for ap-south-1 this week but will keep this thread updated.

[gwryssekk]

@sugi-s ... can you share your appId and region? I'm not having issues with test artifacts in eu-west-1, so I'd like to understand how your tests are configured.

[sugi-s]

@sugi-s Are you seeing this issue since 10/16? I'll have the team investigate either way but please let us know.
@litwicki we started seeing this issue since we integrated test phase to our pipeline which was around 15th Oct

[sugi-s]

@sugi-s ... can you share your appId and region? I'm not having issues with test artifacts in eu-west-1, so I'd like to understand how your tests are configured.

@gwryssekk
appId: d1n442ck0l1wax
region: eu-west-1
concerned branch is in preview (PR) (pr-108.d1n442ck0l1wax)

[gcphost]

Our PRs started randomly failing today, we did not make any changes to our amplify config, code base was passing earlier.

• us-west-2#/d3akotsuqtmpig/pr-1017/3
• d3akotsuqtmpig/pr-1019/1

[litwicki]

@gcphost us-west-2 is being updated today as I type this and will be done before EOD. We'll follow-up here.

[x5hwuk7nc6jsudcc]

We are experiencing the issue in ap-southeast-2, is there any information on when this region will be updated?

[gwryssekk]

@sugi-s -- still investigating your situation. All my attempts with tests have been succeeding so I need to understand the difference. Do you use testcafe for the other branches that succeed?

[Roee-Malis]

We started having similar issues 2-3 days ago.
All linked branches and branch previews are getting stuck at as if the "deploy" stage is in progress.
Deploy logs show: [INFO]: Finish deploying test artifact.
Canceling the build and starting manually does not help.

We are running Cypress E2E tests during the test phase, and it used to work just fine a few days ago.
We did not make any changes to our amplify config.

App id: d19adm9khpw574
Region: us-east-1

Help would be much appreciated.

[sugi-s]

@sugi-s -- still investigating your situation. All my attempts with tests have been succeeding so I need to understand the difference. Do you use testcafe for the other branches that succeed?
@gwryssekk we are running same set of testcafe tests in develop as part of build phase and it works.....

[litwicki]

Hey all, just sharing an update that we're hoping to be wrapped up with all regions in the next day or two. We continue to monitor and are not observing any issues. @Roee-Malis @sugi-s we're continuing to investigate, but your issues do not appear related to deployments but a different issue that is manifesting incorrectly at the deployment step. We are actively investigating.

[Roee-Malis]

@litwicki Thanks for the update. The issue my team was facing was taken care of by AWS support and is no longer recurring as of yesterday.

[x5hwuk7nc6jsudcc]

Deploy issues are resolved for us on ap-southeast-2 with no change to our code or buildspec

[litwicki]

We have now resolved all our deployments through all regions and are confident this issue is resolved. At this time we are closing this issue and in the event of any deployment issue encourage customers to create a new issue.