Aws-sdk-js: DynamoDB client unexpectedly returns BigInt

[kindofbeard]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone through Developer Guide and API reference
• [x] I've checked AWS Forums and StackOverflow for answers
• [x] I've searched for previous similar issues and didn't find any solution

Describe the bug
Starting from AWS-SDK v2.610.0, DynamoDB DocumentClient converts large numbers to BigInt before returning them. Existing Node12 Lambda starts failing with the following TypeError exception:

TypeError: Cannot mix BigInt and other types, use explicit conversions

In Javascript, BigInt and Number types are not compatible. A simple "BigInt(42) + Number(43)" generates a "TypeError" expression. This problem is especially important with Lambda functions, where the AWS-SDK is always set to the latest version. One of my Lambda function running in production starting crashing with "TypeError" messages, even though the code was unchanged.

Is the issue in the browser/Node.js?
Node.js

If on Node.js, are you running this on AWS Lambda?
Yes (with Node 12 runtime)

Details of the browser/Node.js version
v12.14.1

SDK version number
v2.631.0 (Lambda Node.js 12)

• For browsers, the SDK version number is in the script tag

  src=".../aws-sdk-2.466.0.min.js"

• For Node.js, get SDK version by

  • running command npm list aws-sdk from your root directory
  • printing the output of console.log(AWS.VERSION) in your code where AWS = require("aws-sdk");
  • if running on Lambda and using SDK provided by Lambda runtime, you can find the SDK versions here

To Reproduce (observed behavior)

Write one small int and one very large int to DynamoDB, read them back, then use them in the same operation (example: a division).
Run the following code on any DynamoDB table that has string hash key. With latest AWS-SDK, it fails with a TypeError exception:

```
'use strict'

const testRegression = async () => {
const AWS = require('aws-sdk')
const client = new AWS.DynamoDB.DocumentClient()

const TABLE_NAME = 'my-nice-table' // replace with your table
const HASH_KEY = 'hashKey' // replace with your hash key
const SUM_KEY = 'sum'
const COUNT_KEY = 'count'
const hugeInt = 10579824222094800
const smallInt = 42

console.log('Writing two ints, one small (count), one large (sum)')
const params = {
TableName: TABLE_NAME,
Key: { [HASH_KEY]: 'myHash' },
UpdateExpression: set #sum = :hugeInt , #count = :smallInt,
ExpressionAttributeNames: {
'#sum': SUM_KEY,
'#count': COUNT_KEY,
},
ExpressionAttributeValues: {
':smallInt': smallInt,
':hugeInt': hugeInt,
},
}
await client.update(params).promise()

console.log('Read back, then calculate average')
const { Items } = await client
.query({
TableName: TABLE_NAME,
ExpressionAttributeNames: {
'#hashKey': HASH_KEY,
},
ExpressionAttributeValues: {
':hashValue': 'myHash',
},
KeyConditionExpression: #hashKey = :hashValue,
})
.promise()

// Following division fails, because "sum" is a BigInt whereas "count" is not
console.log('Average:', Items[0].sum / Items[0].count)
}

testRegression()
````

Expected behavior
DynamoDB client should not cast returned values to BigInt, unless explicitly requested.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
The regression was introduced in PR #3019, where BigInt support was introduced. Problem lies in "convertNumber" function, under lib/dynamodb/converter.js. The following return statement was added:

return Number.isSafeInteger(numberValue) ? numberValue : BigInt(value);

[ajredniwja]

Hey @kindofbeard thank-you for reaching out to us with your issue, that is potentially a bug let me run some tests and discuss it with the team.

[AllanZhengYP]

@kindofbeard Thank you for informing us the regression. I agree this would introduce breaking change to the customers, even though otherwise SDK might decrease the precision from the BigInt response. I will work on a rollback.

Now I don't think there's a viable way to support BigInt in V2 SDK without breaking customers. We will provide BigInt support in our next major version V3 instead.

[ajredniwja]

Fixed in #3254

Closing this now, please reach out if you have any additional questions.

[TechTeam12]

@AllanFly120 do you have an ETA for when the V3 will be ready for production use?

As an aside, I strongly disagree with removing the BigInt. If someone is doing unsafe math with numbers in node then that is a bug on their side.
The original behavior of a number was to return a string casted as a number when retrieved from the database. This is a dangerous behavior when working with BigInts.

I understand that this would cause people to update their code and fix bugs, and people need to have as little production outage as possible. In the mean time, I'll handle the code on my side and avoid using the DocumentDB client.

[kindofbeard]

@TechTeam12 My main concern here is that existing Lambdas (with bugs inside) start crashing without having been touched. An unsafe number operation is a thing, a crash is another.

Could we go for a { bigInt: true } option or something like that, passed to the DynamoDB SDK instance? Clients that can handle BigInt would then be able to voluntarily opt-in for them ; old clients would keep running without crashing. It would make everyone happy imho.

[TechTeam12]

@kindofbeard moving this to v3 is probably the best solution, depending on timeline. Gamma state should mean that release is imminent.

V2 is in a stable state, and without the warning being included or some kind of optional parameter it probably shouldn't have been merged. https://github.com/aws/aws-sdk-js/issues/3065 Was filed, but not worked on. Adding a configuration to the DocumentClientDB constructor to permit new BigInt behavior, vs old behavior could've been a solution too.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.