Aws-sdk-js: Glacier uploadArchive does not accept ReadableStream in body

[pnxs]

Environment: node.js v0.10.33
AWS-SDK: 2.0.28

I'm trying to upload an archive with "uploadArchive" using a ReadableStream in the "body" parameter:

this.uploadFile = function(filename, done) {
    var stream = fs.createReadStream(filename);

    var params = {
       vaultName: self.vault_name,
       archiveDescription: filename,
       body: stream
    };

    glacier.uploadArchive(params, function(err, data) {
        if (err) {
            console.log("Error uploading archive!", err);
        }
        done();
    });
}

It returns "Error uploading archive! [TypeError: Cannot read property 'length' of undefined]".

Uploading a Buffer or a string works, but i did not want to load the entire file into RAM.

[lsegal]

Due to the way tree hashes are computed in Glacier, we don't yet support streams via uploadArchive because they are not rewindable. I would recommend looking into initiateMultipartUpload(), uploadMultipartPart(), and completeMultipartUpload() (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Glacier.html#initiateMultipartUpload-property). These also don't support streams but you can read from your stream into smaller buffered chunks and upload those instead.

I will mark this as a feature suggestion to add support for streams in uploadArchive and keep this open to track it, but note that it may not be possible to fully support streams due to the lack of rewind on those streamed objects. We could special case files, but not streams in general.

[secretfader]

@lsegal I just ran into this issue too. However, according to the documentation, uploadArchive seems to support streams.

From the docs:

var params = {
  accountId: 'STRING_VALUE', /* required */
  vaultName: 'STRING_VALUE', /* required */
  archiveDescription: 'STRING_VALUE',
  body: new Buffer('...') || 'STRING_VALUE' || streamObject,
  checksum: 'STRING_VALUE'
};
glacier.uploadArchive(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log(data);           // successful response
});

Is this incorrect? The error I keep getting is TypeError: Cannot read property 'length' of undefined, which only occurs when I pass a stream to the body key.

[lsegal]

@nicholaswyoung indeed, streams are not supported as noted above. The documentation is automatically generated based on the type, which technically can accept a stream in the general case, but will not work in the specific case of Glacier (because it requires a customization on top of the core functionality to work). Note however that _all_ streams must have a way to determine their length-- this requirement is the same for all uses of streams in the SDK, not just Glacier. S3 has this restriction as well, and you would likely get the same error if you passed that stream to s3.putObject(), for example.

The SDK looks for a .length or .size property on the object to determine the size, or, as a special case for file streams, does a stat on the file. If the stream has no determinable length, you will get an error. This is the error you are likely getting, not the one related to lack of Glacier support. In other words, even if we added support for streams in Glacier, it would still come with the above restriction and therefore the same above error for streams of undetermined length.

Hope that helps.

[secretfader]

@lsegal I see your reasoning, but the documentation should be correct, whether auto generated or not. Should I open a ticket for this?

Frankly, it's these sorts of issues that make the SDK difficult to rely on, and instead be forced into pkgcloud or other libraries that wrap the official SDK. It's a pain to deal with directly, because so few of the interfaces are written in idiomatic Node-style.

You might want to take a look at how other modules in the same category (namely pkgcloud and Knox) handle basic tasks and try to emulate their style.

I certainly know this oddness has caused me to look at Google's cold storage offerings, because it was overly difficult to build with the SDK.

[lsegal]

@nicholaswyoung I agree that the documentation should be correct. You can open a ticket for this. We would have to add a custom documentation note for Glacier to point out the lack of support.

Frankly, it's these sorts of issues that make the SDK difficult to rely on, and instead be forced into pkgcloud or other libraries that wrap the official SDK. It's a pain to deal with directly, because so few of the interfaces are written in idiomatic Node-style.

You mentioned issues in plural. Can you elaborate on this? We love to get constructive feedback about the SDK. Note that our prioritization and roadmap is heavily based on this kind of user feedback, so if you're running into issues with the library we want to hear about it.

[frankthelen]

Is there any progress on this issue?

[mateodelnorte]

Would also love to see streaming implemented in node.

[agilecollab]

This has also caused me some trouble, after reading the glacier doc:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Glacier.html#uploadArchive-property

Which lead me to believe that uploadArchive() would support streams, ran into problems in my implementation, then found this issue.

The documentation should at least note that glacier's implementation of uploadArchive does not support objectStream. If this documentation noted this quirk it could have saved me a couple of hours today!

That said, I would love to see this as a feature in the future if the requirement of a checksum can be worked around.

[schester44]

+1, some info on how to stream files to Glacier would be awesome. Taking a crack at it using the post above by @lsegal

[github-actions[bot]]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.