Aws-sdk-js: S3 putObject progress

[refaelos]

Is it possible to get the upload (or putObject) progress?

If there was a putStream it might be easier to get it.

[lsegal]

We're looking to add an 'httpProgress' event in light of the new 2.0 release we just made that added browser support to the SDK-- it will definitely be useful there as well.

In Node.js environments you can upload a Stream object and attach a 'readable' event that should get triggered as contents get uploaded. You can potentially track progress there, though you will have to dig into the stream to get the actual amount of data sent over the wire.

[refaelos]

Yeah I'm talking about Node.

Waiting for 2.0 an hoping to get a usage example of any kind.

Thanka

[opensaurusrex]

I would like to see the httpProgress event listener added before official release I have been using a system that sends to PHP and then PHP SDK up to S3 but it's very inefficient that way and we have experienced a few issues with uploading and would also like to keep the progress bar implementation.

[lsegal]

If the goal is only to use this in Node, it's already possible to attach a handler to the stream object you are uploading and track progress like that, as mentioned above. This does not require any changes in the SDK; you can do the following with most versions of the SDK (using the through package to simplify the piping interface):

var fs = require('fs');
var through = require('through');
var numBytes = 0;

var stream = fs.createReadStream('bigfile');
stream.pipe(through(function(data) {
  this.queue(data);
  numBytes += data.length;
  console.log(numBytes, 'bytes written');
}));

var AWS = require('aws-sdk');
var s3 = new AWS.S3({params: {Bucket: 'mybucket'}});
s3.putObject({Key: 'bigfile', Body: stream}).send();

[opensaurusrex]

Will there be an option to do so in the browser SDK?

[lsegal]

@deth4uall I have a WIP branch called http-progress with the above change that adds an httpUploadProgress and httpDownloadProgress event. The event emits a structure that contains the .loaded and .total properties.

Currently this only works in the browser. This will be merged when it is ported over to work inside of Node.js as well.

[lsegal]

PR #209 is now merged and httpUploadProgress / httpDownloadProgress events will be available for use in the next release (or immediately if you build/use from the master branch on GitHub). See description in #209 for example usage.

The feature is available for both the browser and Node.js, but note that only Node.js v0.10.x+ is supported.

[dhiraj72]

@lsegal I am trying to upload a file on S3 by stream using s3.putObject() .I have also configured AWS by setting { accessKeyId": "**_", "secretAccessKey": "_***", "region": "eu-west-1"} and created bucket ,testuser, in eu-west-1 region of S3 . But I got following error. In error It show region:'us-east-1'. What will be solution for this.Thanks in advance .

[NetworkingError: Hostname/IP doesn't match certificate's altnames]
  message: 'Hostname/IP doesn\'t match certificate\'s altnames',
  code: 'NetworkingError',
  region: 'us-east-1',
  hostname: 'testuser.testuser.s3-eu-west-1.amazonaws.com',
  retryable: true,
  time: Thu Jul 17 2014 16:24:44 GMT+0530 (IST),
  statusCode: undefined }

[lsegal]

@dhiraj72 what version of the SDK are you using? Is this in Node.js or the browser?

[dhiraj72]

Hi Isegal ,
Thanks for your response. I am using "aws-sdk" version "2.0.0-rc9" in node.js.

[lsegal]

I would recommend upgrading to the latest version of the SDK: v2.0.8

[dhiraj72]

Thanks @lsegal !!!
After updating AWS SDK It's working for me . :

[john-aws]

Hi, is there any way to get httpUploadProgress events in the browser if you call putObject via makeUnauthenticatedRequest? If I add .on('httpUploadProgress', f) then I see "TypeError: undefined is not a function", for example:

var params = { Key: key, Body: body };

s3.makeUnauthenticatedRequest('putObject', params, function(err, data) {
  console.log("err: " + err);
})
.on('httpUploadProgress', function(progress) { // TypeError here
  console.log("progress: " + progress);
});

[sebastientromp]

Just in case someone else arrives here looking for this:

The makeUnauthenticatedRequest function returns the request object if you don't pass a callback:

return callback ? request.send(callback) : request;

So if you want to add a progress listener, you will do:

var params = { Key: key, Body: body };

var req = s3.makeUnauthenticatedRequest('putObject', params);
req.on('httpUploadProgress', function(progress) {
  console.log("progress: " + progress);
});
req,send(function(err, data) {
  console.log("err: " + err);
})

[kaarthikeyansp]

But I could see only {"isTrusted":true} in progress object now, in 2.94 version of aws-sdk

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.