Bull: How to integrate bull on downloading file to server.

[Rajivkumar1234]

I have a code below that downloads file to the server. I wanted to use bull redis based queue to do the job but I cant find the right way of doing it based on the documentation. Any help would be appreaciated. Thank you.

Code

            var download = function (url, dest, callback) {

                request.get(url)
                    .on('error', function (err) { console.log(err) })
                    .pipe(fs.createWriteStream(dest))
                    .on('close', callback);

            };

            final_list.forEach(function (str) {
                var filename = str.split('/').pop();

                console.log('Downloading ' + filename);

                download(str, filename, function () { console.log('Finished Downloading' + "" + filename) });
            });

[stansv]

var Queue = new Queue("my-queue", "redis://127.0.0.1:6379/0");

queue.add({final_list: final_list});

queue.process(function processor(job, done) {

  var final_list = job.data.final_list;
  var promises = [];

  final_list.forEach(function (str) {
    var filename = str.split('/').pop();
    console.log('Downloading ' + filename);
    promises.push(new Promise(function p(resolve, reject) {
        download(str, filename, function () {
          console.log('Finished Downloading' + "" + filename);
          resolve(filename);
        }); // add error callback also, call reject() on error
      }
    ));
  });

  Promise.all(promises)
    .then(function d(filenames) {
      done(null, filenames);
    })
    .catch(function e(error) {
      done(error);
    })
});

[Rajivkumar1234]

I will try this one Sir, thank you.

[Rajivkumar1234]

@stansv

why this line has been removed

var download = function (url, dest, callback) {

            request.get(url)
                .on('error', function (err) { console.log(err) })
                .pipe(fs.createWriteStream(dest))
                .on('close', callback);

        };

[stansv]

@Rajivkumar1234 I did not tried to provide ready 100% working solution, it is just proposal to help you think in right direction.

[Rajivkumar1234]

I see

[Rajivkumar1234]

But I tried your solution and it works @stansv

[Rajivkumar1234]

Sir can you please explain what is the difference with my previous code vs with the code with the use of bull ? what are the things that have been improved.

[Rajivkumar1234]

@stansv

[stansv]

@Rajivkumar1234

Bull is used for reliable asynchronous, background processing via message queue based on Redis. It means that you want to have an ability to submit time- and memory-consuming, complex tasks very fast, but don't want to get results of each as fast as possible; instead, you want them to be reliably processed in future, have an ability to track each of these tasks to get its status and results when they'll be ready. These tasks can be added to queue on one machine and processed by another one, or a cluster of machines, depending on your needs.

https://en.wikipedia.org/wiki/Message_queue