Aws-sdk-js: How to handle _hardError exceptions from s3.copyObject

[Meekohi]

When using s3.copyObject, if the original object doesn't exist then aws-sdk throws an exception "NoSuchKey: The specified key does not exist.". I'd like to handle these gracefully, but try/catch blocks don't work since the the exception is thrown outside the call stack.

Is there a correct way to handle this? Is it required to explicitly check the existence of the source object before calling s3.copyObject?

[lsegal]

You should never need to handle hard errors explicitly. Can you provide code that causes an error for you?

[Meekohi]

@lsegal I've created the following small example that throws this error. I'm using node v0.10.26 and aws-sdk 2.0.0-rc.17.

https://gist.github.com/Meekohi/088886081deb828f4b4e

[lsegal]

You should never call throw from inside of an asynchronous callback. Weird things happen when you throw exceptions from async function callbacks, as you're no longer guaranteed to be in the same call stack. The only reason the SDK bubbles these exceptions up is so that errors like reference errors (a typo in a variable name or syntax error) don't get swallowed up, but exception handling in these callbacks should not be used for user generated exceptions.

The right way to handle errors is to provide callbacks to the calling function (the Node.js style err callback), or to use domains. Note that you _can_ also use the 'uncaughtException' event, but by Node.js's own documentation, it is effectively deprecated in favor of domains.

Hope that helps. Given that the SDK is behaving as expected I will mark this as closed. Feel free to re-open if you think the SDK is doing something wrong.

[seankovacs]

Why was this closed? aws-sdk is raising an exception bypassing any event emitters or err callbacks if a key doesn't exist. Resorting to a global "catchall" event seems wrong.

[jayzelenkov]

I run into the same exact issue.

Trying to download an object from s3 that doesn't exist is a _programmer error_. There is nothing that could be done about it. That's why aws-sdk throws an uncaughtException at this point.

Network issue would produce an _operational error_ that will be passed to the stream. Then it is up a developer to do something about it:

s3.getObject(params)
      .createReadStream()
      .pipe(file)
      .on('error', (err) => {
        console.log('download failed')
        // retry download ?
      })

One way to avoid uncaughtException is to check if an object exists in s3 bucket before trying to download it.

---

A good article covering errors in NodeJS: https://www.joyent.com/developers/node/design/errors

[seankovacs]

That's odd, fs doesn't crash an app if a file doesn't exist. It actually uses the err in the callback. http doesn't crash when you get on an URL that doesn't exist... interesting.

\_(ツ)_/¯

[jayzelenkov]

good point. APIs are hard

[logidelic]

Is this being looked at? Is there a workaround? To be clear, the problem is that this exception cannot be caught by the client code. This a fatal API bug. No API should throw an exception internally that is not caught internally. This will, at worst, simply crash the node app or, at best, will trigger a global exception handler to allow clean shutdown. Both of these are unacceptable for obvious reasons.

FWIW, I'm getting this issue with s3.getObject when an invalid key name is specified.

[verveguy]

@logidelic: I'm with you on this one.

@lsegal this is insane. I'm in lambda using node and cannot catch this exception. S3 is literally killing my function _with no way for me to gracefully recover_.

To the point earlier, fs doesn't crash your app if a file doesn't exist. That's what exceptions are _for_.

[chrisradek]

@verveguy
Can you share your example where an uncatchable error is being thrown?

Looking at @jzelenkov's example, adding an error event listener to the read stream would catch errors thrown by the service, such as NoSuchKey if trying to download an object that doesn't exist.

[verveguy]

@chrisradek sure.

Note: I now realize that I have _not_ explicitly tried to add event handlers to the _read_ stream. But here's the thing, I shouldn't have to IMHO. In my case, even more so because I'm trying to use the Promise support offered by the AWS SDK.

So what am I doing?

I'm writing a simple (!) lamdba function in node and using the AWS S3 SDK.
Since node is all about async, I'm therefore using Promises. I'm also trying to use the stream support to easily get the file content from S3 onto the /tmp filestore.

What _should_ happen (IMHO) is that by using a Promise, my catch() portion of the promise chain should in fact be invoked with whatever exception is thrown by the machinery underlying S3.getObject . However, that's not the case. S3.getObject() wants to be "special" despite the promise of the promise() support in the AWS SDK. As soon as I call s3.getObject().promise() we're off to the races and the exception is thrown pretty promptly. Note that the promise() is wrapping the AWSRequest, _not_ wrapping the readStream as you propose doing with event handlers.

In the end, I've managed to cobble together a solution that wraps things in a Promise of my own and uses the event handlers on the _output file_ at the end of the pipe to invoke the resolve() , reject().

However, that's too far down the pipe (so to speak) to catch the errors thrown from the event handlers underlying the S3 request. You're right: perhaps I could use the readStream instead. I might even try that sometime.

But for now, I put calls to headObject in place first to avoid even trying to make a getObject call at all since it was unsafe to do so.

Having read various threads around here in growing frustration, this all appears to boil down to a fundamental philosophical argument about what the behavior of an async operation from S3 should be like vs what other async operations (such as fs calls) are like. The (broken) promise support only makes it even more frustrating. And trying to use promises plus streams is making me insane.

I'd love a clean code example of a properly promisified S3.getObject with streams writing to a local file that works without exception even when the key doesn't exist.

[lsegal]

@verveguy can you show a full example of what you are doing? It's hard to know where a problem might be without knowing how to reproduce the issue.

[chrisradek]

How are you attempting to use s3.getObject with both promises and streams? The two aren't currently supported together.

Using just promises, the catch handler on your promise chain should capture the error.

Part of the underlying problem with combining streams and promises in the same interface is that promises only really care about 2 states: resolve and reject. Streams have more than 2 states though, such as data/readable, as well as error and end.

What we would need to do is wrap the current stream that we return through the SDK with a new type that exposes promise methods. We'd still have to deal with the question of whether calling then on the stream will pass the full content as a buffer to the resolving function or not. Since streams are often used to pipe data from one source to another in a memory-efficient way, keeping an internal store of all the data so that it can be returned when using promises would be a huge negative for many users.

It seems like converting streams to promises is something that should be handled outside of the SDK as it would be an important choice made by the user. There are 3rd party libraries that will do just that.

As for the argument for not having to add event handlers to catch errors on the readstream, that's how many event emitter-based APIs work in node.js. Even calling fs.createReadStream with an invalid path will throw an uncatchable error if you do a simple try/catch, and requires the user to attach an error listener to the stream in order to handle the error. Maybe I'm missing something about this argument, please elaborate if so.

@lsegal beat me to the question. Can you share an example of what you're doing? It could be that there really is something the SDK is doing wrong.

[lsegal]

It should also be mentioned, just for clarity, that this _specific_ issue regarded an explicit throw(err) call from inside a request callback (equivalent to throwing from a .then() callback), which is expected to bubble up. The original reported issue was not about errors bubbling out of the SDK's internal handling, which is why it was closed. If you are seeing internal errors bubbling up as thrown errors, it would be great to provide reproduction steps so those scenarios can be looked at.

[verveguy]

@lsegal: I'll open a new issue regarding the use of s3.getObject, streams and promises.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.