Amplify-ios: DataStore sync throws foreign key constraint errors

[dyang]

Describe the bug
My iOS app uses DataStore for local data storage. I notice that whenever there's a need to do a full-sync, e.g. when a user logs in, or when the schemas are changed during development, the sync process might throw some "FOREIGN KEY constraint failed" errors. As a result, no data gets displayed in the app. Users have to terminate the app and re-launch it, then the sync seems to work.

To Reproduce

In my case, I do:

• sign out, which runs DataStore.clear to clear local user data
• sign back in, and then runs DataStore.start to kick off a full-sync

I can reproduce this 100% with my app.

Observed Behavior

No data gets displayed. Neither the syncQueriesReady event nor the ready event get sent.

Expected Behavior

Sync completes without the above errors and send out the ready event(s). (I have some code that displays spinners when there's a potentially long sync)

Stack Trace

Here's what an error might look like:

2021-01-02 10:31:16.186053+0100 App [11981:8265777] [StateMachine<State, Action>] Notifying: errored(DataStoreError: The operation couldn’t be completed. (SQLite.Result error 0.)
Recovery suggestion: The operation couldn’t be completed. (SQLite.Result error 0.)
Caused by:
FOREIGN KEY constraint failed (code: 19))

Upon closer inspection, I find that the error was thrown at SQLiteStorageEngineAdapter.save method. Whenever this happens, it's because the failing object has a foreign key dependency on another object. In my case I have a Model called Episode, which depends on another object Media. The simplified model schema looks like this:

type Episode @model @auth(rules: [{ allow: owner}]) {
  id: ID!
  title: String!
  media: Media! @connection
}

type Media @model @auth(rules: [{ allow: owner}]) {
  id: ID!
  audioUrl: String
  s3Key: String
  mimeType: String
}

Most of the time the corresponding media record gets saved before the sync engine attempts to save its containing episode, but this is not always the case. I put a break point at the catch clause in the above mentioned save method, and when the foreign key error happens, I run a query against the local sqlite db with the media id, and the media record is not there.

It's interesting that later in the sync process, that "missing" media record does eventually get saved, so this looks more like a race condition than data lost.

Areas of the SDK you are using (AWSMobileClient, Cognito, Pinpoint, IoT, etc)?

DataStore, Storage, Authentication

Environment(please complete the following information):

• SDK Version: 1.5.2
• Dependency Manager: Cocoapods
• Swift Version : 5.0
• Xcode Version: 12.2

Device Information (please complete the following information):

• Device: iPhone 12, Simulator]
• iOS Version: iOS 14.3

[dyang]

Update: Even though I'm not losing any data from the backend, data sync does seem to miss some items from time to time. In a DynamoDB table I have less than 6 items, but sometimes data sync only manages to sync 2 or 3 items down to the local database. This happened quite a few times during the last two days.

[anatomybook]

I'm seeing a similar (possibly the same) issue.

According to my investigation as well, the foreign key constraint failure is due to a data race. The problem (at least for me) shows in the console as
[ReconcileAndLocalSaveOperation] The operation couldn’t be completed. (Amplify.DataStoreError error 8.)

What clearly "helps" for me as well, is to sync an empty local db against a filled, i.e. several thousand items, remote counterpart, as @dyang reported.

Here is what I investigated so far:

Suppose the data structure given above, the initial sync builds two objects of type AWSModelReconciliationQueue, one for Media and one for Episode. Each of these has its own private let reconcileAndSaveQueue: OperationQueue.
During initial sync, these queues are filled with objects of type ReconcileAndLocalSaveOperation.
In AWSInitialSyncOrchestrator.enqueueSyncableModels the sync is triggered in the correct dependency order, however, it is possible that one model queue "overtakes" the other. More specifically:

As far as I can tell, it's possible that a queue item for Episode sits for a long time in state .waiting if the Episode queue is not prioritized. This may lead to queue Media "overtaking" queue Episode attempting to locally insert Media items without their Episode (still in state ReconcileAndLocalSaveOperation.waiting), triggering foreign key failures.

Does this make any sense?

Please let me know if you need any other information.

I've touched this some time ago already in issue https://github.com/aws-amplify/amplify-ios/issues/773 but back in the day, I didn't know that https://github.com/aws-amplify/amplify-ios/issues/773 actually dealt with two separate issues. Anyhow, there are also two console dumps, where I captured that the child item is written before the parent.

[anatomybook]

Just a side-note: About a week ago, I "fixed" my local installation by moving the OperationQueue com.amazonaws.DataStore.\(modelSchema.name).reconcile up one level, i.e. from being a member of AWSModelReconciliationQueue to be being a member of AWSIncomingEventReconciliationQueue, now renamed plain com.amazonaws.DataStore.reconcile. I.e., objects of type AWSModelReconciliationQueue now receive their OperationQueue as an argument from the outside instead of making their own.
For me, this has fixed or at least improved the issue to the point that I haven't seen it since. The rationale was to have only one queue, so different type items cannot overtake each other anymore. Seems to work fine for me, as I have synced my database from scratch many times until today.
Notice, however, I explicitly don't claim that this is a fix for the problem. I've yet to fully comprehend all the queues and threads deep inside DataStore. So there's probably better ways to fix this. But, at least for me, it helped. So I figure, I'd share, after I tried it out for a while...

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. Please, provide an update or it will be automatically closed in 7 days.

[ruiguoamz]

One of my findings:

1.
Using schema with @connectionbelow:

type Episode @model {
  id: ID!
  title: String!
  media: Media! @connection
}

type Media @model {
  id: ID!
  audioUrl: String
  s3Key: String
  mimeType: String
}

and with the implementation of moving ReconcileAndLocalSave up a level to be part of AWSIncomingReconciliationQueue(Only one OperationQueue for two Model), the Initial Sync takes about 9~10 seconds to finish.

2.
Using schema without @connection below:

type Episode @model {
  id: ID!
  title: String!
  media: String!
}

type Media @model {
  id: ID!
  audioUrl: String
  s3Key: String
  mimeType: String
}

and with the current DataStore implementation (AWSModelReconciliationQueue for each Model), the Initial Sync takes about 8~9 seconds to finish.

Haven't tried with more complex schema.

[anatomybook]

Hello @ruiguoamz ,

thanks for looking into this. Do you think it's plausible that the above mentioned problems may arise from one queue overtaking the other? Or am I completely off?

I haven't exactly timed the original code vs. my modification, but ,subjectively, it is not much slower, matching your observation.

Thanks!

[anatomybook]

Is there any additional information, I may be able to supply?

[ruiguoamz]

@dyang @anatomybook
The PR with serial ReconcilieAndLocalSave solution was merged but not yet released.

Will notify when we finish the release

[anatomybook]

@ruiguoamz
Sounds great. Thanks for the update.

[dyang]

@ruiguoamz Awesome! Thanks for the fix. 👍

[lawmicha]

This was released and subsequent fix for it was released in 1.7.2. Please use 1.7.2 and newer