Nodejs-storage: Frequent 408 Timeout errors while uploading multiple images in parallel

Created on 10 Dec 2020 · 6Comments · Source: googleapis/nodejs-storage

Hi all,
We are having frequent 408 timeout errors due to which it has become unusable for us.

@google-cloud/storage : "^5.5.0"
node: 12-alpine
Avg Image Size: ~5MB

We are running 3 docker containers of same application to upload multiple images in parallel. Randomly gcs stucks for >5 minutes and then return 408 Timeout error.

Code

await bucket("bucketName")
      .upload(filename, {
        destination: "testDestination",
        gzip: true,
        metadata: {
          cacheControl: "no-cache"
        },
        resumable: false
      })

We have also tried file.createWriteStream but having same issue.

Error

error while completing job Multiple errors occurred during the request. Please see the `errors` array for complete details.

    1. Request Timeout
    2. <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 408 (Request Timeout)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>408.</b> <ins>That’s an error.</ins>
  <p>Your client has taken too long to issue its request.  <ins>That’s all we know.</ins>

We really appreciate any help in this regard.
Thanks

storage question

Source

muhammadbilal87

👍1

Most helpful comment

We have already gone through issue #1281, we were still facing the same problem so we created a new issue.

muhammadbilal87 on 15 Dec 2020

👍2

All 6 comments

Hi @muhammadbilal87 , does this workaround resolve the issue? If so, this might be related to https://github.com/googleapis/nodejs-storage/issues/1281

shaffeeullah on 14 Dec 2020

We have already gone through issue #1281, we were still facing the same problem so we created a new issue.

muhammadbilal87 on 15 Dec 2020

👍2

Im encountering this error as well @shaffeeullah, My use case is slightly different, I am reading in streams from cloud storage and zipping the streams using a stream archiver, and uploading the resulting stream with createWriteStream. Small file sizes work great. But larger ones throw this error. I have also tried setting the timeout option but that does not help either. Any help would be appreciated.

cor1 on 29 Jan 2021

Hi @muhammadbilal87 and @cor1 ,

Thank you for reporting this issue. I am trying to replicate it by uploading multiple 16mb images in parallel, but the requests are succeeding. 408 errors often occur when the server thinks there is an unused connection. Is it possible that uploading multiple large images in parallel is straining your internet bandwidth or causing connection issues? Is your traffic spiking or is this a steady workload and this is happening out of the blue? The error message says Your client has taken too long to issue its request; I suspect that the server is not seeing a request at all. Also, a fix for #1281 has been released. If you update to the latest version of the library, are you still seeing this issue?

shaffeeullah on 29 Jan 2021

@shaffeeullah , I just checked that I was on the latest version but still encountered issues. I thought it could be my internet connection as well, so i put the code on a 4gb ram cloud run instance and encountered the same issue. For me it happened out of the blue. It was working fine when i did around a 100 mb upload, but when it got above that it started giving me issues. the request goes for around 7 minutes before throwing the 408 error. I am strictly using streams, and this upload was the only one happening at the time of testing.

cor1 on 29 Jan 2021

@shaffeeullah , quick update. I refactored some of my code in the way that it was handling streams, and removed a passthrough, this seems to have fixed my specific issue. I am no longer receiving the error locally or on cloud run. I will let you know if my specific issue starts happening again, but I think i solved it for now! thanks for you help!

cor1 on 30 Jan 2021

🎉1

Was this page helpful?

0 / 5 - 0 ratings