Dynamoose: Version 2.0 Rewrite

[fishcharlie]

Version 2.0 Rewrite

Summary:

This issue tracks the full rewrite of Dynamoose for version 2.0.

Last Steps Before Final:

• [ ] After deployment we need to make sure the cache control is correct for the items

Next Steps:

• [x] Rethink dynamoose.undefined based on comments in #726
• [ ] Model.update
  
  
  
  • [ ] Add support for set if_not_exists
  
  
  • [ ] Add support for setting to not update updatedAt timestamp when updating
  
  
• [ ] Ensure support for enum for lists, and sets works correctly
  
  
  
  • [ ] Ensure this works for Model.update when both setting, and appending to an array or set
  
  
• [ ] errorUnknown Schema setting from Dynamoose v1
• [ ] Model.delete update option
• [ ] Figure out allowEmptyArray setting for Model.update
• [ ] Figure out createRequired setting for Model.update
• [ ] Figure out returnValues setting for Model.update
• [ ] Add support for only getting certain attributes in Model.get (options.attributes)
• [ ] Add support for consistent option in Model.get (options.consistent) (ensure this also exists on Scan/Query)
• [ ] Do research on if Indexes can exist in nested properties (inside lists or maps) (I don't think they can). If not we should truly think about if they should exists on the Schema or if we should move them to the Model. If we decide to keep it on the Schema, we should throw an error for nested schemas if the nested schema contains an index.
• [ ] Rethink about if there is a better way to handle expires/ttl other than adding a property to the Schema
• [ ] Figure out how to remove utils/empty_function.js from code coverage summary after npm test
• [x] Clean up system for finding index and rearranging parameter names for Query
• [ ] Add support for splitting up batchGet requests for every 100 items and making multiple requests
• [ ] Add support for processing UnprocessedKeys in batchGet
• [ ] Do more research between global and local indexes
• [ ] Think about if rangeKey & hashKey should truly be on the Schema like v1, since it only gets set per Table, maybe it should be on the Model.

---

Long term steps:

• [ ] Allow for custom updateDetails timeout for updateTimeToLive. We should also maybe refactor how the setup flow works. We are doing a bunch of different things, and in a lot of cases we retrieve data before running a check function. We should standardize that process to make the code more dry.
• [ ] There is an issue where when we do JSON.stringify(test.schema) throughout the test suite, it prints {}, we need to figure out how to fix this

v2.x Tasks:

• [x] TypeScript Support (#788)
• [ ] Plugin support
• [ ] Model.populate
  
  
  
  • It'd be cool for Model.populate to support passing another Model in as the type of a Schema property, that would basically create a reference to that other Model.
  
  
• [ ] Nested Schema support (#301)
• [ ] https://github.com/dynamoosejs/dynamoose/issues/768#issuecomment-608160537
• [ ] Add support for null DynamoDB type
• [ ] Add support for passing in array of conditions into function settings (like Model.update)
• [ ] Add support for custom date types like moment
• [ ] Add support for ISO date format
• [ ] Add support for custom large numbers like bignumber.js
• [ ] Add support for setting tags on the table

Goals:

• 100% code coverage with automated tests that cover as many edge cases as can possibly be thought of
• Better documentation, everything should be documented as it's being written and created
• Fix a lot of pesky long lingering bugs in the codebase (ex. #669)
• Better parity with Mongoose
• Taking advantage of ES6 and beyond features from top to bottom
  
  
  
  • This is not always possible. Especially when it comes to classes. https://stackoverflow.com/q/58864709/894067
  
  
• Have the codebase be more sustainable to adopt more features and functionality in the future, for example things like TypeScript, and new JavaScript features

Other Notes:

• I assume that v2.0 will be a step backwards from Dynamoose v1.x at the beginning, for example I don't believe the following items don't make it into v2.0:
  
  
  
  • Plugins
  
  
  • Populate
  
  
  • TypeScript type file
  
  
• This description will change frequently as work is done on this project, so check back often for updates
• v2.0 will include MAJOR breaking changes, it will be likely that we will encourage all users to test their entire Dynamoose stack from top to bottom due to the amount of breaking changes that will be introduced

News/Announcements:

• February 16th 2020 - Beta 1 of Version 2.0 has been released. I'm really looking for a lot of feedback as devs integrate the beta version. If you have a smooth or difficult experience, I want to hear about it all. Feel free to comment below or contact me.
• November 14th 2019 - We are looking for submissions of unit or integration tests to be run on features or functionality to be prioritized for version 2.0. These tests should be very through and cover as many edge cases as possible. If you have reported bugs in Dynamoose v1.x that still haven't been resolved (or have), this is a great opportunity to have your issues (or features) be prioritized for v2.0. In order to participate, please submit a pull request with your tests added. This PR should be branched off of and merged into the 2.0 branch.

[hardyscc]

Thank you for your excellent library, I has been using the version 2.0 and created the
NestJS Dynamoose Module. I also created a demo Serverless Starter Project which utilized those libraries. Thank for your great work again, it make my life so much easier :)

[MickL]

@fishcharlie Great work! Really looking forward for v2 and will try it out together with the NestJS module :) What do you think about using TypeScript decorators for everything? Like TypeORM or dynamo-types does? I wonder why v2 is not made in TypeScript in general? :)

@hardyscc I just started a new Nest app with DynamoDB and saw your comment randomly, thats awesome! Could you try to get you module into the Nest docs for all others to come?

[hardyscc]

@MickL I just released this nest module in very short time to fulfil my need, still have many missing pieces. I am waiting Dynamoose 2.0 to come out first and will definitely notify the nest community soon. thanks :)

[fishcharlie]

Thank you for your excellent library, I has been using the version 2.0 and created the
NestJS Dynamoose Module. I also created a demo Serverless Starter Project which utilized those libraries. Thank for your great work again, it make my life so much easier :)

@hardyscc Awesome!!! Glad it worked well for you. If you have any other feedback or suggestions, please let me know.

[fishcharlie]

@fishcharlie Great work! Really looking forward for v2 and will try it out together with the NestJS module :) What do you think about using TypeScript decorators for everything? Like TypeORM or dynamo-types does? I wonder why v2 is not made in TypeScript in general? :)

@MickL I'm the main contributor to Dynamoose at this point, and I have never used TypeScript to date. It is on my list of things to work on. Maintainability is a massive priority for Dynamoose v2. I'm unwilling at this point to jump into a new technology on this project when I don't feel comfortable and confident with it. I believe Dynamoose v2 really prepares for an easy migration to refactoring to use TypeScript sometime soon. It just comes down to my level of prioritization, and I have chosen to prioritize things like documentation, test coverage, bug fixes, and code maintainability, over TypeScript. I believe that is reflected in the work I have done with v2.

As mentioned in the Goals section above Have the codebase be more sustainable to adopt more features and functionality in the future, for example things like TypeScript, and new JavaScript features. So TypeScript is 100% on my radar, and others have mentioned this as well. I'm confident it'll happen at some point (hopefully soon), but I don't know when and can't really give an ETA at this point.

Hopefully that helps clear up that question and gives some insight into my thinking regarding this.

[MickL]

Sounds good and I can totally understand especially when it comes to documentation, test coverage and bug fixes.

[fishcharlie]

https://github.com/dynamoose/dynamoose/releases/tag/v2.0.0