Dynamoose: Index gets re-created on every restart without good reason

[dolsem]

Summary:

Here's the issue I spent a long time solving. I delete the table and restart my app. The table gets created instantly along with the email GSI, but the requestIndexKey GSI is stuck on "Creating" in the DynamoDB console for several minutes (sometimes up to 10 minutes). In some cases the app crashes as a result of a timeout. In other cases it completes before the timeout and I can use the table just fine. Regardless, at some point I restart the app and the issue gets worse: if update is set to false the app simply crashes saying that the index is not synchronized, if update is set to true the same behavior repeats itself (long wait or crash).

Finally I was able to identify the source of the issue, kudos to @williscool and his work on #440. The updates happen because the order of items in the projected attributes array matters when remote and local indexes are compared, while the order of items for remote seems to depend on the order of attributes in the table (either that or alphabetical). So if you list attributes in a different order, the index constantly gets re-created. I fixed the order and got rid of the problem.

Code sample:

Schema

const UserSchema = new dynamoose.Schema({
  id: {
    type: String,
    lowercase: true,
    hashKey: true,
  },
  password: {
    type: String,
    minlength: 6,
  },
  avatar: {
    type: String,
    validate: /https:\/\/(\w+:{0,1}\w*@)?(\S+)(:[0-9]+)?(\/|\/([\w#!:.?+=&%@!\-/]))?/,
  },
  email: {
    type: String,
    required: true,
    validate: /^\w+([.-]?\w+)*@\w+([.-]?\w+)*(\.\w{2,3})+$/,
    index: {
      global: true,
      project: ['password'],
    },
  },
  firstName: {
    type: String,
    required: true,
  },
  lastName: {
    type: String,
    required: true,
  },
  requestCount: {
    type: Number,
  },
  requestIndexKey: {
    type: String,
    set: ([requestCount, username]) => `${BIG_NUMBER - requestCount}.${username}`,
    index: {
      global: true,
      project: ['avatar', 'firstName', 'lastName'],
    },
  },
}, { // UserSchema options.
  useDocumentTypes: true,
  timestamps: true,
});

Model

const User = dynamoose.model(MODEL_NAME, UserSchema, { update: true });

Debug output

  dynamoose:table local:  { IndexName: 'requestIndexKeyGlobalIndex',
  KeySchema: [ { AttributeName: 'requestIndexKey', KeyType: 'HASH' } ],
  Projection: 
   { ProjectionType: 'INCLUDE',
     NonKeyAttributes: [ 'avatar', 'firstName', 'lastName' ] } } +0ms
  dynamoose:table remote:  { IndexName: 'requestIndexKeyGlobalIndex',
  KeySchema: [ { AttributeName: 'requestIndexKey', KeyType: 'HASH' } ],
  Projection: 
   { ProjectionType: 'INCLUDE',
     NonKeyAttributes: [ 'firstName', 'lastName', 'avatar' ] } } +1ms

Current output and behavior:

Index is re-created on every restart if there's mismatch in the order of elements.

Expected output and behavior:

The order of elements should not be the cause of the index update.

Environment:

Operating System: Linux
Operating System Version: 4.15.0-36-generic
Node.js version (node -v): v9.11.2
NPM version: (npm -v): 6.4.1
Dynamoose version: 1.1.0

Type (select 1):

• [x] Bug report
• [ ] Feature suggestion
• [ ] Question
• [ ] Other suggestion
• [ ] Something not listed here

Other:

• [x] I have read through the Dynamoose documentation before posting this issue
• [x] I have searched through the GitHub issues (including closed issues) and pull requests to ensure this issue has not already been raised before
• [x] I have searched the internet and Stack Overflow to ensure this issue hasn't been raised or answered before
• [x] I have tested the code provided and am confident it doesn't work as intended
• [x] I have filled out all fields above
• [x] I am running the latest version of Dynamoose

[fishcharlie]

@dolsem Thanks for reporting! I have a few questions so I can work on debugging this.

• Does this problem only occur when using the actual AWS DynamoDB service? Or does problem occur when using local DynamoDB as well?
• You said Finally I was able to identify the source of the issue and I fixed the order and got rid of the problem. Can that easily be converted into a PR? If so, is there a reason we can't do that? If not, do you mind providing more information about where you would able to identify the source in the code so I can take a look at that area specifically?

Sounds like this might just be a matter of adding some type of sort to both array's or something before comparing them.

---

In terms of the 10 minutes it takes to create that index. I don't really see that being something we can really fix. The time to create an index depends on the number of items in your table, and is really reliant on AWS. If you have any ideas of how to improve that system I'd love to hear them, but I'm not sure if there is anything we can do to improve that.

---

I'd love to get this fixed for you as soon as possible so let me know what I can do to help with this to get this solved quickly for you.

[dolsem]

Sorry for being unclear. I haven't looked at the Dynamoose code yet. The issue I identified and solved was in my own code. I simply changed the order of attributes in that array in my schema to the order in the array that comes from remote. Unfortunately, I likely won't have time in the next two weeks to help solve this problem in Dynamoose.

Yes, this happened when I connected to the AWS service. I haven't tried to reproduce this issue with a local instance.

I've done all of the testing with an empty table, so the number of items could not have affected the time the index gets created. Besides, the email index gets created in a matter of seconds, so it's very strange that it takes that long to create the requestIndexKey index. I understand that the issue is most likely with DynamoDB itself, but I wanted to report it in case it's somehow relevant. I haven't seen this behavior before. Also, the time it takes the app to restart before it can work with the database again + plus potential timeout, after which the app has to be restarted again, all due to this long index creation, are the reason I had to look closely into what's happening and solve it in my code.

[fishcharlie]

@dolsem Ok, I will see if I can find some time to reproduce this and try to see what is going on.

[dolsem]

@fishcharlie also, it seems like after I change the order of items so that it matches remote, index creation becomes very fast.

[fishcharlie]

@dolsem What do you mean by changing the order of items?

[dolsem]

@fishcharlie I use debug output to look at the order of items in the projected attributes array of the index that comes from remote. I change the array in my schema definition, so that the order is the same.

[fishcharlie]

@dolsem I believe I have fixed this problem in PR #455. Can you please test that PR before I merge it in and let me know if it fixes your problem or not?

Also, we need to get some tests written for this PR. Kinda struggling conceptualizing how to do that. Any input on that would be really awesome too.

[dolsem]

@fishcharlie okay, I will test it and see if I can capture the problem with a test. But I'll need several days before I can get to it. Thanks for looking into this!

[fishcharlie]

@dolsem No problem! Just keep me posted. Hopefully that fixes it for you.

[fishcharlie]

@dolsem Any chance you get around to testing this?