Cli: 502 Bad Gateway when deploying large site

[unwitting]

- Do you want to request a feature or report a bug?

Bug, sort of. We might be stretching what's classed as supported behaviour, in which case I'm requesting that that scope be widened to sites of this size.

- What is the current behavior?

When attempting to deploy a large site (upwards of 20000, sometimes 50000 total files) via the netlify CLI, we often see the following error either during the _CDN diffing files_ phase or (as in this case) the _Uploading files_ phase:

Deploying to live site URL...
- Hashing files...
✔ Finished hashing 57892 files
- CDN diffing files...
✔ CDN requesting 29854 files
- Uploading 29854 files
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at NetlifyAPI.uploadDeployFile (/usr/local/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)
    at <anonymous>
Exited with code 2

This is from a CI server environment, so one with a (presumably) pretty reliable and fast internet connection.

When run on a local machine, we get slightly more overview of the behaviour. The upload appears to work as expected right up until the end:

Deploying to live site URL...
✔ Finished hashing 57893 files
✔ CDN requesting 30907 files
⠋ (30906/30907) Uploading s/404.html...
 ›   Warning: TextHTTPError: 502 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
⠒ (30906/30907) Uploading s/404.html...TextHTTPError: Bad Gateway
    at NetlifyAPI.uploadDeployFile (~/.nvm/versions/node/v8.11.4/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)
    at <anonymous>

You can see above, the tool claims the bad gateway on the very last file (which in this case happens to be 404.html). Re-running the deploy at this point usually results in the CDN saying it doesn't need any of the files uploading again, and successfully finishing.

- If the current behavior is a bug, please provide the steps to reproduce.

I think reproduction steps might be to generate a site with around 50000 files and try to deploy it. I can try to form a simple Gatsby setup that might generate 50000 unique-looking files on each build if needed.

- What is the expected behavior?

A successful deploy.

[jtraulle]

I have the same issue for a very large site :

✔ Finished hashing 810170 files
- CDN diffing files...

 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at NetlifyAPI.createSiteDeploy (~/.nvm/versions/node/v11.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)

[unwitting]

Holy wow, 810k files 😅

[bwlng]

We have the same issue on a site w/ 5477 files. We generally get the Bad Gateway error the first time we run netlify deploy and then run the command again and the build runs without issue.

Deploying to draft URL...
✔ Finished hashing 5477 files
⧇ CDN diffing files... ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
⧆ CDN diffing files...TextHTTPError: Bad Gateway
    at NetlifyAPI.createSiteDeploy (~/.nvm/versions/node/v10.16.1/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)

[laurenskling]

I tried to upload a whopping 1.6m files, only got errors. Reached out to the support team, they said they would not be able to handle such a large site. They would only be able to handle about 40k files sites.

[devpatelwebdeveloper]

We are getting the same issue at 4000 files. Sometimes, it runs smooth with no issue and sometimes it keeps on failing whole day.

Deploying to live site URL...
- Hashing files...
 ✔️ Finished hashing 4496 files
- CDN diffing files...
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 › TextHTTPError: Bad Gateway at parseResponse (/usr/src/node_modules/netlifycli/node_modules/netlify/src/methods/response.js:9:11)
error Command failed with exit code 2.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

script returned exit code 1

[mkrn]

This is happening consistently on a website with about 2450 files, 1200 pages.
Curious if there's a way to deploy zip file using netlify deploy?

Command:

netlify deploy $DEPLOYFLAG --auth $NETLIFY_TOKEN --site=$NETLIFY_SITE_ID --json

› Warning: TextHTTPError: 502 › Warning: › TextHTTPError: Bad Gateway › TextHTTPError: Bad Gateway at parseResponse (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:9:11) at async callMethod (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:38:26) at async module.exports (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/deploy/index.js:78:16) at async NetlifyAPI.deploy (/usr/lib/node_modules/netlify-cli/node_modules/netlify/src/index.js:92:12) at async DeployCommand.run (/usr/lib/node_modules/netlify-cli/src/commands/deploy.js:158:17) at async DeployCommand._run (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:44:20) at async Config.runCommand (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:160:9) at async Main.run (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:21:9) at async Main._run (/usr/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:44:20)

[lukehsiao]

I am also seeing this issue on a site with just over 4100 pages. However, I'm also not able to deploy manually with the drag/drop upload, meaning we are currently stuck and unable to deploy our site.

[devpatelwebdeveloper]

Netlify has open the issue https://github.com/netlify/cli/issues/745 for this problem, but its medium priority

[xmflsct]

Same here with 4k files.

[thandaanda]

We are having issues with 1000 files. Also, is there any way to check which particular file is failing?

[stephencweiss]

I'm on the low end of site size here (~4800 pages), and noticed this just started happening:

$ netlify deploy
Deploy path: ~/code/blog/public
Deploying to draft URL...
✔ Finished hashing 4799 files
⊷ CDN diffing files... ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›

I got around it eventually by manually setting a 10 minute timeout: netlify deploy --timeout 6000

$ netlify deploy --timeout 6000 --message "with a timeout"
Deploy path: ~/code/blog/public
Deploying to draft URL...
✔ Finished hashing 4799 files✔ CDN requesting 2417 files
✔ Finished uploading 2417 assets✔ Draft deploy is live!

Not an ideal solution... but hopefully it helps others!
/shrug

[danielgavrilov]

We also started noticing flaky deploys 2 days ago, with all failures happening at the "CDN diffing" stage:

- Hashing files...
✔ Finished hashing 64624 files
- CDN diffing files...
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at parseResponse (~/live/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/live/node_modules/netlify/src/methods/index.js:36:26)
    at async module.exports (~/live/node_modules/netlify/src/deploy/index.js:79:16)
    at async NetlifyAPI.deploy (~/live/node_modules/netlify/src/index.js:93:12)
    at async DeployCommand.run (~/live/node_modules/netlify-cli/src/commands/deploy.js:203:17)
    at async DeployCommand._run (~/live/node_modules/@oclif/command/lib/command.js:44:20)
    at async Config.runCommand (~/live/node_modules/@oclif/config/lib/config.js:151:9)
    at async Main.run (~/live/node_modules/@oclif/command/lib/main.js:21:9)

We upgraded to 2.58.0 yesterday, since there was a fix merged (#966) intended to address #745, which is related to this issue.

We still got another 2 deploy failures today at the “CDN diffing files…” stage, so it seems this is still an issue.

If it helps, our site is fully public: https://github.com/owid/owid-static

We can’t reliably reproduce the issue, but it was occurring between:

• [2020-07-08 08:07:07] successful deploy https://github.com/owid/owid-static/commit/a26c8f546bb83bcd47656ecada5a78781d8978a5
• [2020-07-08 08:16:03] failed deploy (CDN diffing error) https://github.com/owid/owid-static/commit/17737a1ad4063924f50db2320f3951b1c74fe4c4
• [2020-07-08 09:36:47] failed deploy (CDN diffing error) https://github.com/owid/owid-static/commit/b9fb32fb7f6b11170270a9f64351c6b88cca4abb
• [2020-07-08 09:58:40] successful deploy https://github.com/owid/owid-static/commit/c2c5b21604a0ee2b480f45b12a750c8fd05e5224

All the above were running netlify-cli 2.58.0 on a DigitalOcean droplet (London). We always run netlify deploy on the server instead of GitHub-triggered deploys just because it saves us a few minutes.

For now, we've increased the deploy timeout to 10min, as suggested here, and we will update this if we see any more failures.

[ViachaslauBohdan]

Guys do you know whether this kind of error can be related to this issue as well?

`✔ Client-side fallback created: 200.html
ℹ Ready to run nuxt serve or deploy dist/ directory
Deploy path: /root/repo/rgnuxt/dist
Functions path: /root/repo/rgnuxt/functions
Configuration path: /root/repo/rgnuxt/netlify.toml
Deploying to draft URL...

• Hashing files...
  ✔ Finished hashing 29532 files and 1 functions
• CDN diffing files...
  ✔ CDN requesting 19671 files and 0 functions
• Uploading 19671 files
  › Warning: JSONHTTPError: undefined 500
  › Warning:
  › {
  › "name": "JSONHTTPError",
  › "status": 500,
  › "json": {
  › "status": 500,
  › "error": "Internal Server Error"
  › }
  › }
  ›
  JSONHTTPError: Internal Server Error
  at parseResponse (~/repo/rgnuxt/node_modules/netlify/src/methods/response.js:12:11)
  at runMicrotasks ()
  at async callMethod (~/repo/rgnuxt/node_modules/netlify/src/methods/index.js:36:26)`

[Auspicus]

Setting a timeout as @stephencweiss suggested in https://github.com/netlify/cli/issues/484#issuecomment-636211718 worked for me as a temporary workaround.