Graphback: DataSync with ability to persist change-logs

[ssd71]

I am looking at

1. Delta Store: The ability to store ‘edit history’ of individual documents.
2. Server-side Conflict Resolution: Use the same to resolve update conflicts on the server.
3. Delta Queries: Fetch the latest versions of documents that have changed since a given timestamp

Some thoughts into how this might be done on Mongo

1. Delta Store

• One historical collection per Model
• For the historical collection, we could model the data like: create a delta document for every update with:
  
  
  
  • updatedAt, version and updatedAt fields and a _deleted flag that says when the change occurred as well as if the document was deleted in the operation.
  
  
  • docId field that stores the _id of the original document
  
  
  • sets field that contains a map of name of updated field to field value after update.
  
  

2. Server-side Conflict Resolution

• With the above setup, a simple procedure to resolve conflicts in data would be something like:
  
  
  
  1. Issue a findOneAndUpdate command with a filter like { _id: "meh", version: 5, _deleted:{ $ne: true } }, if it succeeds go to (iii).
  
  
  2. At this point, if the command has failed, we either have a conflict on our hands or the document doesn't exist in the database. To differentiate, issue a findOne query with just the _id and _deleted filter. If we get a valid result, the versions do not match. We then proceed to check if the field that is being updated, has already been updated between version:5 and the current state, this will be done by making use of the delta table. If it has, we reject the update and abort the procedure. If it has not, we bump the version in the update document then retry from (i)
  
  
  3. Add a delta document to the historical collection that corresponds to the update that just happened
  
  

3. Delta Queries

We can already issue delta queries with what is currently on master.

[wtrocki]

Lets continue updating this thread. Pinned it to the repo

[ssd71]

Delta Queries: Use the same to fetch what fields changed since a given timestamp.

This goes to the bin after #1744 , gonna edit that out

[ssd71]

@wtrocki I would like to propose something else for the Server-side conflict resolution: current strategy is to use findOneAndUpdate then use findOne to check if there are conflicts, resolve conflicts then try again

I would like to propose the following:

• use findOne to get the document in the database
• check for conflicts, resolve conflicts
• try to update with findOneAndUpdate, if failed, restart the procedure

Why: Previous procedure is only suitable for _version conflicts, but proposed procedure can be modified to suit any conflict detection and resolution

Catch: Even if there are no conflicts, we have to make at least two database calls

(I probably haven't thought through everything)

[wtrocki]

yep. You have described how this works. Yo avoid loop we need to repeat this max x times. x can be 2 or 3 or whatever

[ssd71]

Completed in #1833