Nodejs-storage: Out of memory when using readStream on big files

[trample25]

Hello,
I tried to download 50GB File from Google Cloud Platform to my local machine.
I used GCP's getReadStream function, and I saw that the memory usage was increasing dramatically until the process crashed.

What is the best practice for using your readStream? Or is there any workaround for that problem?

Thanks!

[stephenplusplus]

Hey @trample25. What version of our library are you using? And would you mind sharing the minimum code required to reproduce the issue? (You can assume I have a 50GB file to work with)

[trample25]

Hi @stephenplusplus, thank you for your quick response.
I'm using version 2.0.0

const { createWriteStream } = require('fs-extra');
const {Storage} = require('@google-cloud/storage');
async function main() {
    const storage = await getStorage();
    const bucket= 'my bucket';
    const testFile1 = '50GBFile';

    const readStream = storage.bucket(bucket).file(testFile1).createReadStream();
    const writeStream = createWriteStream(testFile1);
    readStream.pipe(writeStream);
}
main();

Please monitor the memory usage while performing this test. It seems like the readstream buffers all the data into the memory.

Thanks!

[stephenplusplus]

So far, I haven't been able to reproduce. Watching top, the memory usage sticks around ~50 MB, but that value would likely fluctuate depending on the resources available to your specific machine and process. Node's internals should be figuring all of that out automatically.

We don't buffer data into memory. Instead, it is passed through using normal stream mechanics, where data is only fetched when the destination is ready to receive it. Then when that receiver clears it, it asks for more, we get more, etc.

Can you run and monitor just this script, and paste the output (you can use my bucket and file!)?

const fs = require('fs-extra');
const {Storage} = require('@google-cloud/storage');

const testFile1 = '50GBFile';

const storage = new Storage()
const bucket = storage.bucket('stephen-has-a-new-bucket');
const file = bucket.file(testFile1)

const readStream = file.createReadStream();
const writeStream = fs.createWriteStream(testFile1);

const startingMemoryUsage = getMemoryUsage()
var lastMemoryUsage
var timeLastLogPrinted

function getMemoryUsage() {
  lastMemoryUsage = Math.ceil(process.memoryUsage().heapUsed / 1024 / 1024)
  return lastMemoryUsage
}

function printMemoryUsage() {
  if ((Date.now() - timeLastLogPrinted) < 3000) {
    // Only log every 3 seconds
    return
  }

  if (lastMemoryUsage === getMemoryUsage()) {
    // Nothing new to report
    return
  }

  console.log(`Memory usage relative to starting memory: ${lastMemoryUsage - startingMemoryUsage} MB`)
  timeLastLogPrinted = Date.now()
}

readStream
  .on('data', printMemoryUsage)
  .pipe(writeStream)

My output

$ node . 
Memory Usage: -2 MB
Memory Usage: 3 MB
Memory Usage: 0 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 4 MB
Memory Usage: 3 MB
Memory Usage: -2 MB
Memory Usage: 3 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 4 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 1 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 5 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB

[trample25]

Hi @stephenplusplus, thanks again for your professional and detailed response.
I ran your script and as you can see, the memory usage has increased significantly:
My output:

node script.js
Memory usage relative to starting memory: -3 MB
Memory usage relative to starting memory: 5 MB
Memory usage relative to starting memory: 13 MB
Memory usage relative to starting memory: 16 MB
Memory usage relative to starting memory: 20 MB
Memory usage relative to starting memory: 25 MB
Memory usage relative to starting memory: 17 MB
Memory usage relative to starting memory: 23 MB
Memory usage relative to starting memory: 22 MB
Memory usage relative to starting memory: 23 MB
Memory usage relative to starting memory: 24 MB
Memory usage relative to starting memory: 26 MB
Memory usage relative to starting memory: 10 MB
Memory usage relative to starting memory: 15 MB
Memory usage relative to starting memory: 12 MB
Memory usage relative to starting memory: 21 MB
Memory usage relative to starting memory: 23 MB
Memory usage relative to starting memory: 17 MB
Memory usage relative to starting memory: 21 MB
Memory usage relative to starting memory: 32 MB
Memory usage relative to starting memory: 30 MB
Memory usage relative to starting memory: 34 MB
Memory usage relative to starting memory: 41 MB
Memory usage relative to starting memory: 38 MB
Memory usage relative to starting memory: 35 MB
Memory usage relative to starting memory: 37 MB
Memory usage relative to starting memory: 43 MB
Memory usage relative to starting memory: 49 MB
Memory usage relative to starting memory: 55 MB
Memory usage relative to starting memory: 45 MB
Memory usage relative to starting memory: 57 MB
Memory usage relative to starting memory: 59 MB
Memory usage relative to starting memory: 55 MB
Memory usage relative to starting memory: 60 MB
Memory usage relative to starting memory: 69 MB
Memory usage relative to starting memory: 76 MB
Memory usage relative to starting memory: 68 MB
Memory usage relative to starting memory: 73 MB
Memory usage relative to starting memory: 72 MB
Memory usage relative to starting memory: 75 MB
Memory usage relative to starting memory: 83 MB
Memory usage relative to starting memory: 74 MB
Memory usage relative to starting memory: 77 MB
Memory usage relative to starting memory: 81 MB
Memory usage relative to starting memory: 85 MB
Memory usage relative to starting memory: 91 MB
Memory usage relative to starting memory: 93 MB
Memory usage relative to starting memory: 95 MB
Memory usage relative to starting memory: 97 MB
Memory usage relative to starting memory: 105 MB
Memory usage relative to starting memory: 106 MB
Memory usage relative to starting memory: 107 MB
Memory usage relative to starting memory: 108 MB
Memory usage relative to starting memory: 109 MB
Memory usage relative to starting memory: 94 MB
Memory usage relative to starting memory: 95 MB
Memory usage relative to starting memory: 94 MB
Memory usage relative to starting memory: 95 MB
Memory usage relative to starting memory: 108 MB
Memory usage relative to starting memory: 95 MB
Memory usage relative to starting memory: 108 MB
Memory usage relative to starting memory: 6 MB
Memory usage relative to starting memory: 10 MB
Memory usage relative to starting memory: 18 MB
Memory usage relative to starting memory: 10 MB
Memory usage relative to starting memory: 17 MB
Memory usage relative to starting memory: 18 MB
Memory usage relative to starting memory: 29 MB
Memory usage relative to starting memory: 19 MB
Memory usage relative to starting memory: 21 MB
Memory usage relative to starting memory: 33 MB
Memory usage relative to starting memory: 32 MB
Memory usage relative to starting memory: 33 MB
Memory usage relative to starting memory: 34 MB
Memory usage relative to starting memory: 35 MB
Memory usage relative to starting memory: 41 MB
Memory usage relative to starting memory: 2 MB
Memory usage relative to starting memory: 10 MB
Memory usage relative to starting memory: 8 MB
Memory usage relative to starting memory: 16 MB
Memory usage relative to starting memory: 19 MB
Memory usage relative to starting memory: 21 MB
Memory usage relative to starting memory: 14 MB
Memory usage relative to starting memory: 21 MB
Memory usage relative to starting memory: 22 MB
Memory usage relative to starting memory: 23 MB
Memory usage relative to starting memory: 33 MB
Memory usage relative to starting memory: 36 MB
Memory usage relative to starting memory: 35 MB
Memory usage relative to starting memory: 40 MB
Memory usage relative to starting memory: 41 MB
Memory usage relative to starting memory: 45 MB
Memory usage relative to starting memory: 46 MB
Memory usage relative to starting memory: 45 MB
Memory usage relative to starting memory: 50 MB
Memory usage relative to starting memory: 49 MB
Memory usage relative to starting memory: 50 MB
Memory usage relative to starting memory: 49 MB
Memory usage relative to starting memory: 50 MB
Memory usage relative to starting memory: 58 MB
Memory usage relative to starting memory: 5 MB
Memory usage relative to starting memory: 6 MB
Memory usage relative to starting memory: 10 MB
Memory usage relative to starting memory: 22 MB
Memory usage relative to starting memory: 21 MB
Memory usage relative to starting memory: 17 MB
Memory usage relative to starting memory: 20 MB
Memory usage relative to starting memory: 35 MB
Memory usage relative to starting memory: 37 MB
Memory usage relative to starting memory: 27 MB
Memory usage relative to starting memory: 33 MB
Memory usage relative to starting memory: 41 MB
Memory usage relative to starting memory: 43 MB
Memory usage relative to starting memory: 41 MB
Memory usage relative to starting memory: 45 MB
Memory usage relative to starting memory: 46 MB
Memory usage relative to starting memory: 55 MB
Memory usage relative to starting memory: 54 MB
Memory usage relative to starting memory: 51 MB
Memory usage relative to starting memory: 57 MB
Memory usage relative to starting memory: 60 MB
Memory usage relative to starting memory: 71 MB
Memory usage relative to starting memory: 66 MB
Memory usage relative to starting memory: 63 MB
Memory usage relative to starting memory: 64 MB
Memory usage relative to starting memory: 61 MB
Memory usage relative to starting memory: 62 MB
Memory usage relative to starting memory: 63 MB
Memory usage relative to starting memory: 64 MB 

What do you think is the reason for that behavior?
Thanks!

[stephenplusplus]

Thanks for running that! It seems like the stream internals are actually working. Note the various drops where it goes from 58 MB -> 5 MB, 108 MB -> 6 MB, etc. That would mean the memory usage of your system built up until it needed a breath, and flushed the rest out of the internal buffer, reclaiming that space.

As far as why your memory usage grew to those much higher levels than my test run, that could be due to different factors that differentiate our testing environments. So, I'm not sure I should be reading into that too much.

What would be bad is if that memory usage grew and grew, and never flushed down, but that's not what we're seeing in this test case. So we have to figure out what made my test case different from your initial script that ran until it crashed. Would you be able to create a sample repository that I can clone and run with that code that wasn't releasing memory?

[trample25]

Hi @stephenplusplus, Actually the first crash happened due to download of 5 files of 10GB, then I tried it with 1 file of 50GB and it also crashed (and now I see that sometimes it works fine even though it still uses lots of memory). Unfortunately I can't share the actual code that didn't release memory, but I can reproduce it again easily with a simple script :

for(var i =1; i<11; i++){
    let bucket = 'my-bucket';
    let testFile1 = `file${i}`;
    let testFile2 = `10FilesOf10GB/file${i}`;
    let readStream = storage.bucket(bucket).file(testFile2).createReadStream(options);
    let writeStream = createWriteStream(testFile1);
    readStream.pipe(writeStream);
  }

When I run this code, the memory usage goes to 100% and I get ESOCKETTIMEDOUT error.
If it matters, I use node version 8.12 + run the script on linux machine (Amazon Linux AMI 2018.03.0.20180508 x86_64 HVM )

Thank you very much!

[stephenplusplus]

I couldn't reproduce with 10 concurrent streams downloading 50 GB files each. The memory usage stayed around the same as before. Though, having that many streams running without being aware of each other could be a problem. The file system will also get hit pretty hard with that demand.

In the snippet below, I've introduced async to manage the flow control to 5 parallel streams at once. It will also log the memory usage as a percentage of the available memory. Would you mind running this and reporting back what you find?

Also, how are you monitoring memory usage?

Thanks!

const async = require('async')
const fs = require('fs-extra');
const {Storage} = require('@google-cloud/storage');

const testFile1 = '50GBFile';

const storage = new Storage()
const bucket = storage.bucket('stephen-has-a-new-bucket')
const file = bucket.file(testFile1)

const readStream = file.createReadStream()
const writeStream = fs.createWriteStream(testFile1)

const startingMemoryUsage = getMemoryUsage()
var lastMemoryUsage
var lastAvailableMemoryPercentage
var timeLastLogPrinted

function getMemoryUsage() {
  const memoryUsage = process.memoryUsage()
  lastMemoryUsage = Math.ceil(memoryUsage.heapUsed / 1024 / 1024)
  lastAvailableMemoryPercentage = (memoryUsage.heapTotal / memoryUsage.heapUsed).toFixed(2)
  return lastMemoryUsage
}

function printMemoryUsage() {
  if ((Date.now() - timeLastLogPrinted) < 1000) {
    // Only log ~ once per second
    return
  }

  if (lastMemoryUsage === getMemoryUsage()) {
    // Nothing new to report
    return
  }

  console.log(`
Memory Stats:
  Available memory relative to starting memory: ${lastMemoryUsage - startingMemoryUsage} MB
  Usage relative to available memory: ${lastAvailableMemoryPercentage}% `)

  timeLastLogPrinted = Date.now()
}

// arg1: How many times should this function run
// arg2: How many parallel functions should run at once
async.timesLimit(10, 5, (n, next) => {
  file.createReadStream()
    .on('error', next)
    .on('data', printMemoryUsage)
    .pipe(fs.createWriteStream(`file${n}`))
    .on('error', next)
    .on('finish', next)
}, (err) => {
  if (err) throw err
  console.log('Done downloading!')
})

[trample25]

I'm using the top command in linux in order to monitor the memory usage.
This is what I see at the row of the node process before it gets killed:
(PID)2890 (RES)14g (%MEM)97.7 (COMMAND)node

And this is the output of your latest script (please pay attention to the error in the end):

node script1.js 

Memory Stats:
  Available memory relative to starting memory: 2 MB
  Usage relative to available memory: 1.73% 

Memory Stats:
  Available memory relative to starting memory: 8 MB
  Usage relative to available memory: 2.08% 

Memory Stats:
  Available memory relative to starting memory: 15 MB
  Usage relative to available memory: 1.81% 

Memory Stats:
  Available memory relative to starting memory: 10 MB
  Usage relative to available memory: 2.30% 

Memory Stats:
  Available memory relative to starting memory: 23 MB
  Usage relative to available memory: 1.65% 

Memory Stats:
  Available memory relative to starting memory: 27 MB
  Usage relative to available memory: 1.63% 

Memory Stats:
  Available memory relative to starting memory: 30 MB
  Usage relative to available memory: 1.63% 

Memory Stats:
  Available memory relative to starting memory: 36 MB
  Usage relative to available memory: 1.56% 

Memory Stats:
  Available memory relative to starting memory: 33 MB
  Usage relative to available memory: 1.77% 

Memory Stats:
  Available memory relative to starting memory: 42 MB
  Usage relative to available memory: 1.59% 

Memory Stats:
  Available memory relative to starting memory: 43 MB
  Usage relative to available memory: 1.64% 

Memory Stats:
  Available memory relative to starting memory: 48 MB
  Usage relative to available memory: 1.58% 

Memory Stats:
  Available memory relative to starting memory: 49 MB
  Usage relative to available memory: 1.65% 

Memory Stats:
  Available memory relative to starting memory: 63 MB
  Usage relative to available memory: 1.40% 

Memory Stats:
  Available memory relative to starting memory: 58 MB
  Usage relative to available memory: 1.60% 

Memory Stats:
  Available memory relative to starting memory: 67 MB
  Usage relative to available memory: 1.47% 

Memory Stats:
  Available memory relative to starting memory: 68 MB
  Usage relative to available memory: 1.52% 

Memory Stats:
  Available memory relative to starting memory: 74 MB
  Usage relative to available memory: 1.47% 

Memory Stats:
  Available memory relative to starting memory: 85 MB
  Usage relative to available memory: 1.35% 

Memory Stats:
  Available memory relative to starting memory: 87 MB
  Usage relative to available memory: 1.36% 

Memory Stats:
  Available memory relative to starting memory: 92 MB
  Usage relative to available memory: 1.34% 

Memory Stats:
  Available memory relative to starting memory: 86 MB
  Usage relative to available memory: 1.48% 

Memory Stats:
  Available memory relative to starting memory: 99 MB
  Usage relative to available memory: 1.35% 

Memory Stats:
  Available memory relative to starting memory: 94 MB
  Usage relative to available memory: 1.45% 

Memory Stats:
  Available memory relative to starting memory: 98 MB
  Usage relative to available memory: 1.44% 

Memory Stats:
  Available memory relative to starting memory: 111 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 115 MB
  Usage relative to available memory: 1.30% 

Memory Stats:
  Available memory relative to starting memory: 108 MB
  Usage relative to available memory: 1.42% 

Memory Stats:
  Available memory relative to starting memory: 122 MB
  Usage relative to available memory: 1.30% 

Memory Stats:
  Available memory relative to starting memory: 118 MB
  Usage relative to available memory: 1.36% 

Memory Stats:
  Available memory relative to starting memory: 124 MB
  Usage relative to available memory: 1.33% 

Memory Stats:
  Available memory relative to starting memory: 123 MB
  Usage relative to available memory: 1.36% 

Memory Stats:
  Available memory relative to starting memory: 125 MB
  Usage relative to available memory: 1.37% 

Memory Stats:
  Available memory relative to starting memory: 134 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 130 MB
  Usage relative to available memory: 1.37% 

Memory Stats:
  Available memory relative to starting memory: 140 MB
  Usage relative to available memory: 1.30% 

Memory Stats:
  Available memory relative to starting memory: 137 MB
  Usage relative to available memory: 1.36% 

Memory Stats:
  Available memory relative to starting memory: 146 MB
  Usage relative to available memory: 1.30% 

Memory Stats:
  Available memory relative to starting memory: 143 MB
  Usage relative to available memory: 1.35% 

Memory Stats:
  Available memory relative to starting memory: 149 MB
  Usage relative to available memory: 1.32% 

Memory Stats:
  Available memory relative to starting memory: 159 MB
  Usage relative to available memory: 1.26% 

Memory Stats:
  Available memory relative to starting memory: 151 MB
  Usage relative to available memory: 1.34% 

Memory Stats:
  Available memory relative to starting memory: 158 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 156 MB
  Usage relative to available memory: 1.34% 

Memory Stats:
  Available memory relative to starting memory: 160 MB
  Usage relative to available memory: 1.33% 

Memory Stats:
  Available memory relative to starting memory: 164 MB
  Usage relative to available memory: 1.33% 

Memory Stats:
  Available memory relative to starting memory: 166 MB
  Usage relative to available memory: 1.32% 

Memory Stats:
  Available memory relative to starting memory: 171 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 181 MB
  Usage relative to available memory: 1.25% 

Memory Stats:
  Available memory relative to starting memory: 172 MB
  Usage relative to available memory: 1.33% 

Memory Stats:
  Available memory relative to starting memory: 175 MB
  Usage relative to available memory: 1.33% 

Memory Stats:
  Available memory relative to starting memory: 189 MB
  Usage relative to available memory: 1.25% 

Memory Stats:
  Available memory relative to starting memory: 180 MB
  Usage relative to available memory: 1.33% 

Memory Stats:
  Available memory relative to starting memory: 190 MB
  Usage relative to available memory: 1.28% 

Memory Stats:
  Available memory relative to starting memory: 185 MB
  Usage relative to available memory: 1.32% 

Memory Stats:
  Available memory relative to starting memory: 194 MB
  Usage relative to available memory: 1.28% 

Memory Stats:
  Available memory relative to starting memory: 191 MB
  Usage relative to available memory: 1.32% 

Memory Stats:
  Available memory relative to starting memory: 199 MB
  Usage relative to available memory: 1.28% 

Memory Stats:
  Available memory relative to starting memory: 207 MB
  Usage relative to available memory: 1.24% 

Memory Stats:
  Available memory relative to starting memory: 210 MB
  Usage relative to available memory: 1.24% 

Memory Stats:
  Available memory relative to starting memory: 212 MB
  Usage relative to available memory: 1.25% 

Memory Stats:
  Available memory relative to starting memory: 206 MB
  Usage relative to available memory: 1.30% 

Memory Stats:
  Available memory relative to starting memory: 207 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 208 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 222 MB
  Usage relative to available memory: 1.24% 

Memory Stats:
  Available memory relative to starting memory: 225 MB
  Usage relative to available memory: 1.24% 

Memory Stats:
  Available memory relative to starting memory: 227 MB
  Usage relative to available memory: 1.24% 

Memory Stats:
  Available memory relative to starting memory: 218 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 220 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 224 MB
  Usage relative to available memory: 1.30% 

Memory Stats:
  Available memory relative to starting memory: 225 MB
  Usage relative to available memory: 1.31% 

Memory Stats:
  Available memory relative to starting memory: 239 MB
  Usage relative to available memory: 1.24%      

if (err) throw err
           ^

Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/disk1/data/source/node_modules/request/request.js:816:19)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at ClientRequest.emit (events.js:208:7)
    at TLSSocket.emitTimeout (_http_client.js:707:34)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at TLSSocket.emit (events.js:208:7)
    at TLSSocket.Socket._onTimeout (net.js:422:8)
    at ontimeout (timers.js:498:11) 

The difference between my test and yours might be related to the environment we are using?
What OS do you use and which node version?

[trample25]

Hi @stephenplusplus @kinwa91 , do you have any updates regarding this issue?
Thanks!

[jkwlui]

ping @stephenplusplus

[stephenplusplus]

Sorry for the delay. @trample25, would you mind updating to the latest release, 2.3.4, and running the script again? I have run it, and my results are similar to my other runs-- hovering around -4 to +4 MB memory usage relative to my starting point.

Regarding the ESOCKETTIMEDOUT error, I think that was an unrelated issue that has since been resolved. Let me know if that comes back, though.

• OS: macOS Mojave, 10.14.2
• Node: 10.11.0

[JustinBeckwith]

Greetings @trample25! If you're still running into issues, please do let us know. We're going to close this out for now.

[ivancevich]

Hi guys, I'm running into this issue when using the file.createReadStream method. It seems this problem was introduced in version 2.2.0 and is present in all version since then (including version 2.5.0). I'm using version 2.1.0 to avoid running into this issue.

My specific use case:

• I'm streaming files that are between 100MB and 200MB in size.
• I'm using the node-throttle module.
• I'm piping the stream into the HTTP response of Express.js

bucket
    .file(filename)
    .createReadStream({ start, end })
    .on("error", err => {
        if (err.code === "ESOCKETTIMEDOUT") {
            logger.warn("Error reading file stream", err);
        } else {
            logger.error("Error reading file stream", err);
        }
    })
    .pipe(throttle)
    .pipe(res);

Some measurements I made using the memory-usage module:

• v2.1.0:
  

• v2.2.0:
  

• v2.4.0:
  

• v2.5.0:
  

As you can see, the version 2.1.0 graph drops memory usage, while the later versions increase linearly over time and never release the memory back, even after stopping streaming.