Dynamoose: [BUG] Type Mismatch error message on hashKey when unrelated field is null

[willheslam]

Summary:

Thanks for your great work with dynamoose!

I've been porting over a codebase from dynamoose 1 -> 2, and I noticed a weird issue.

Writing a document with a null field and a schema with saveUnknown: true, then querying it causes a "Type Mismatch' error that complains about the hashKey:
UnhandledPromiseRejectionWarning: TypeMismatch: Expected hashKey to be of type string, instead found type object.

Removing the null field or removing saveUnknown: true causes the query to complete normally.

There's two aspects to this as far as I can tell:

1. It's unclear whether having a field that is null with saveUnknown is a problem
2. The type mismatch error blames a seemingly unrelated hashKey.

I've created a minimal-ish reproduction of the problem, using dynamodb local.
Clone the repo, npm install then run npm test to see the problem.

Code sample:

https://github.com/willheslam/dynamoose-type-mismatch/

Schema

const schema = new dynamoose.Schema(
  {
    hashKey: {
      type: String,
      hashKey: true,
      required: true,
    },
  },
  { saveUnknown: true }
)

Model

const MyModel = dynamoose.model("mytable", schema, modelOptions)

General

    await MyModel.batchPut([
      {
        hashKey: "foo",
        bar: null, // comment this out and the type mismatch goes away
      },
    ])
    const records = await MyModel.query({ hashKey: { eq: "foo" } }).exec()

Current output and behavior (including stack trace):

(node:25257) UnhandledPromiseRejectionWarning: TypeMismatch: Expected hashKey to be of type string, instead found type object.
    at checkTypeFunction (/Users/will.heslam/recreate-dynamoose-bug/node_modules/dynamoose/lib/Document.ts:263:11)
    at Array.map (<anonymous>)
    at Function.<anonymous> (/Users/will.heslam/recreate-dynamoose-bug/node_modules/dynamoose/lib/Document.ts:282:111)
    at Generator.next (<anonymous>)
    at /Users/will.heslam/recreate-dynamoose-bug/node_modules/dynamoose/dist/Document.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/will.heslam/recreate-dynamoose-bug/node_modules/dynamoose/dist/Document.js:4:12)
    at Function.Document.objectFromSchema (/Users/will.heslam/recreate-dynamoose-bug/node_modules/dynamoose/dist/Document.js:192:12)
    at Document.<anonymous> (/Users/will.heslam/recreate-dynamoose-bug/node_modules/dynamoose/lib/Document.ts:421:40)
    at Generator.next (<anonymous>)

Expected output and behavior:

records [
  Document { hashKey: 'foo', foo: null },
  lastKey: undefined,
  count: 1,
  queriedCount: undefined,
  timesQueried: 1
]

Environment:

Operating System:
Operating System Version:
Node.js version (node -v): v12.14.1
NPM version: (npm -v): 6.13.4
Dynamoose version: 2.1.2

Other information (if applicable):

Other:

• [x] I have read through the Dynamoose documentation before posting this issue
• [x] I have searched through the GitHub issues (including closed issues) and pull requests to ensure this issue has not already been raised before
• [x] I have searched the internet and Stack Overflow to ensure this issue hasn't been raised or answered before
• [x] I have tested the code provided and am confident it doesn't work as intended
• [x] I have filled out all fields above
• [x] I am running the latest version of Dynamoose

[fishcharlie]

@willheslam This is actually intentional. It is in preparation for supporting the DynamoDB null type in the future.

You can find this in the breaking change log:

Trying to save a Document with a property set to null will now throw an error. If you would like to remove the property set it to dynamoose.UNDEFINED to set it to undefined without taking into account the default setting, or undefined to set it to undefined while taking into account the default setting.

It is kinda a pain I know. But I felt like this was a worthwhile tradeoff to be able to support the native DynamoDB null type in the future.

[willheslam]

Cheers @fishcharlie - reading fail on my part then! :( Sorry about that.

I think the thing that threw me was the error message complaining about the hashKey being of type string, instead of explicitly saying that nulls aren't supported yet. I'm assuming that's a bug?

[fishcharlie]

@willheslam

I think the thing that threw me was the error message complaining about the hashKey being of type string, instead of explicitly saying that nulls aren't supported yet. I'm assuming that's a bug?

Woah yeah. Glanced right past that. That is a super weird error message. Should be more clear. Will add status:todo to this and will try to reproduce and look into it when I can. Might be low priority tho.

[MickL]

I am also having a very very weird error when using Model.create(): TypeMismatch: Expected date to be of type number, instead found type object.

I am processing tons of items from another API so i cant find out what exactly the problem is. Invested couple of hours already :(

What is super weird:

• date is always a number, I didnt found any item where it isnt
• I get UnhandledPromiseRejectionWarning even tho I catch ALL calls i do with Dynamoose
• After that error I cant fetch my table with .scan() anymore, it gives me a TypeMismatch, too: UnhandledPromiseRejectionWarning: TypeMismatch: Expected date to be of type number, instead found type object.

I am also using saveUnknown for this schema...#

EDIT: Found it! It was a totally different attribute that was null, not date and there was nothing object instead of number as the error message stated. So my conclusion:

• Wrong error if something is null (object instead of X)
• Wrong reference if something is null (wrong attribute name)
• Uncatched promise somehow

Man took me hours and SO much frustration :( THANKS @williscool without this issue I would have never found it...

[fishcharlie]

@willheslam Finally got some time to look into this. The issue here is that you are using null. Which does lead to strange behavior as I mentioned above. The reason that the error message complains that hashKey is an object instead of a string, is because internally the object it receives is {"hashKey": {"S": "foo"}, "bar": {"NULL": true}} (which is a DynamoDB object). Now normally, when a Document is created, it checks to see if it's a DynamoDB object, and converts it to a standard object. However, in this case, Dynamoose thinks that is not a DynamoDB object. When you run Document.isDynamoObject({"id": {"S": "foo" }, "data": {"NULL": true}}) it returns false. When in reality it should return true.

The reason for that is part of the isDynamoObject has this code:

const {Schema} = require("./Schema");
const attributeType = Schema.attributeTypes.findDynamoDBType(key);
return typeof value === "object" && keys.length === 1 && attributeType && (nestedResult || Object.keys(value[key]).length === 0 || attributeType.isSet);

In this case when it looks at the null property, attributeType is false. So it assumes the entire object is not a DynamoDB object, when in reality, it is.

The solution for this is just to add support for NULL attribute types in Dynamoose. I've opened an issue for that #951.

[github-actions[bot]]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 3 days.

[fishcharlie]

@willheslam I forgot to ask this. What is your use case here? When do you use null as the property value vs just having the property not exist?

[willheslam]

@fishcharlie Our use case is communicating with a third party that stores its own state.
Sending a property will update that property on their end, no property at all means 'leave as is', and a value of null will mean 'delete that property'.

Another similar use case would be that in JSON you can't represent { foo: undefined }, but you can represent {} and { foo: null }.

Thanks for looking into this and discovering the root cause!

[alexpovh]

@willheslam @fishcharlie Thank you guys for sharing your findings!

I'm facing the same issue and in my case I can't find a way to delete a field via $SET at all. Using $REMOVE works as a workaround though. This applies to updating a field that is not part of the static schema with saveUnknown option enabled for the schema.

WIth $SET I tried the following without luck:

1. Pass undefined as a field value: field does not get deleted
2. Pass null as a field value: field is replaced with a Null type and this makes subsequent reads fail (described above in this thread)
3. Pass dynamoose.UNDEFINED as a field value: getting the following error in this case

          "TypeError: String cannot represent value: Symbol(dynamoose.undefined)",
          "    at GraphQLScalarType.serializeString [as serialize] (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/type/scalars.js:159:9)",
          "    at completeLeafValue (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:635:37)",
          "    at completeValue (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:579:12)",
          "    at completeValueCatchingError (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:495:19)",
          "    at resolveField (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:435:10)",
          "    at executeFields (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:275:18)",
          "    at collectAndExecuteSubfields (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:713:10)",
          "    at completeObjectValue (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:703:10)",
          "    at completeValue (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:591:12)",
          "    at completeValueCatchingError (/Users/alexey-no-backup/no-backup/work/git/app-experience-registry/node_modules/graphql/execution/execute.js:495:19)"

[softmarshmallow]

https://github.com/dynamoose/dynamoose/issues/815

saving scene Document { id: 'Ej4ZE0vrGRohG6k4_a_Ed', projectId: 'temp', fileId: '', nodeId: '6671:952', sdkVersion: 'v2020.0', designPlatform: 'com.figma.Desktop', cachedPreview: '', sceneType: 'SCREEN', route: Symbol(dynamoose.undefined), name: 'Frame 1333', description: Symbol(dynamoose.undefined), tags: [], alias: Symbol(dynamoose.undefined), variant: Symbol(dynamoose.undefined), layers: [ { nodeId: 'I6671:933;2074:140553;2074:140542', index: 3, name: 'profile/customer/40', sdkVersion: 'v2020.0', data: [Object], type: 'VANILLA', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 44, height: 44, x: 48, y: 70 }, { nodeId: 'I6671:933;2074:140553;2074:140542', index: 1003, name: '(bg-converted) profile/customer/40', sdkVersion: 'v2020.0', data: [Object], type: 'RECT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 44, height: 44, x: 48, y: 70 }, { nodeId: 'I6671:933;2074:140553;2074:140543', index: 4, name: 'customer/{name}', sdkVersion: 'v2020.0', data: [Object], type: 'TEXT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 227, height: 21, x: 100, y: 81.5 }, { nodeId: 'I6671:933;2074:140553', index: 1002, name: '(bg-converted) segments/customer/ (mini)', sdkVersion: 'v2020.0', data: [Object], type: 'RECT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 279, height: 44, x: 48, y: 70 }, { nodeId: 'I6671:933;1902:11729;1875:118413', index: 4, name: 'Frame 325', sdkVersion: 'v2020.0', data: [Object], type: 'VANILLA', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 279, height: 8, x: 48, y: 130 }, { nodeId: 'I6671:933;1902:11729;1875:118413', index: 1004, name: '(bg-converted) Frame 325', sdkVersion: 'v2020.0', data: [Object], type: 'RECT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 279, height: 8, x: 48, y: 130 }, { nodeId: 'I6671:933;1902:11729;1875:118416', index: 5, name: 'segments/customer/status/customer-steady', sdkVersion: 'v2020.0', data: [Object], type: 'TEXT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 92, height: 16, x: 48, y: 144 }, { nodeId: 'I6671:933;1902:11729', index: 1003, name: '(bg-converted) segments/customer/status/customer-steady', sdkVersion: 'v2020.0', data: [Object], type: 'RECT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 279, height: 30, x: 48, y: 130 }, { nodeId: 'I6671:933;1314:480', index: 4, name: 'Date of Issue : 2020/05/10', sdkVersion: 'v2020.0', data: [Object], type: 'TEXT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 143, height: 14, x: 48, y: 176 }, { nodeId: '6671:933', index: 1001, name: '(bg-converted) rows/kit-customer', sdkVersion: 'v2020.0', data: [Object], type: 'RECT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 327, height: 168, x: 24, y: 46 }, { nodeId: '6671:952', index: null, name: '(bg-converted) Frame 1333', sdkVersion: 'v2020.0', data: [Object], type: 'RECT', layers: Symbol(dynamoose.undefined), componentId: Symbol(dynamoose.undefined), width: 375, height: 298, x: 0, y: 0 } ], width: 375, height: 298 } [Nest] 51547 - 12/07/2020, 11:30:08 AM [ExceptionsHandler] Cannot convert undefined or null to object +67ms TypeError: Cannot convert undefined or null to object at Function.entries (<anonymous>) at Schema.getTypePaths (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:377:17) at /Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:446:34 at Array.reduce (<anonymous>) at Schema.getTypePaths (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:377:33) at /Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:446:34 at Array.reduce (<anonymous>) at Schema.getTypePaths (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:377:33) at /Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:446:34 at Array.reduce (<anonymous>) at Schema.getTypePaths (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Schema.ts:377:33) at /Users//Documents/GitHub/services/node_modules/dynamoose/lib/Model/index.ts:417:81 at Array.map (<anonymous>) at Model.schemaForObject (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Model/index.ts:417:58) at Function.Document.prepareForObjectFromSchema (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Document.ts:200:38) at Document.toDynamo (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Document.ts:491:17) at Document.save (/Users//Documents/GitHub/services/node_modules/dynamoose/lib/Document.ts:142:30) at ScenesService.<anonymous> (/Users//Documents/GitHub/services/scene-store-service/src/scenes/scenes.service.ts:51:35)

Replaced all null / undefined to dynamoose.UNDEFINED, still getting the same error.