Amplify-js: Datastore Feedback

[vlac666]

Hi, Following on from #5764 . I would like to add some feedback and suggestions for Datastore.

Before I feedback on Datastore I would like to say that I only started using Appsync because the Amplify documentation says it has great offline integration with Apollo It is a little frustrating to hear that the strategy has changed but the guidance for offline scenarios has not !!!

My feedback

1. Does a record exist or not? My single biggest issue working with Datastore is that you can't trust it when it says a record does not exist. Sat in a fully connected office, near zero latency. When I issue a query for a specific record that I know exists. The first query to Datastore (on a clean machine) always returns that the record does not exist.

I was really keen to use Datastore but when I noticed that the results of a read could not be relied upon I had to drop Datastore. My application can't have records behaving like weird quantum particles. When I am connected I need certainty not probability.

1. Have I got everything now? I didn't give up there. I thought I still had good use cases for Datastore, take the classic meter reading route. Download a route to the device and then walk around collecting meter readings. Sync up when you get connectivity. Clear the device datastore ready for the next user. Perfect for Datastore ?

a. What are my routes today - none!! Hang-subscriptions are firing. Have I got all of them now?
b. Has all of the route data been transferred yet? Am I safe to start the route or will the first calling point arrive after I have set off?
c. When I have completed the routes - how do I know all of the data has synced back? I can't clear the datastore until it has synced.

1. I really hate loosing simple graphql queries to get back structured objects.

My honest opinion is that it takes too much control and feedback away from developers.

My Suggestions

1. It's great that you sync things for us and have taken, the hard work away but we also need control over the sync queries - trust us to know what else we might want to sync.
2. We need to be able to trigger a sync and know when syncs have completed - trust us to know when we want to sync and give us the feedback we need to let users know they are good to go.
3. Give us a read through option or at least give us feedback so we know that object type is still syncing.
4. When you have done all that - Try building a real world use case and try it in the real world. Until then , please, please get appsync working properly with latest Apollo client as a viable offline solution (just like the docs recommend).

I can see where you are trying to get with it and I can see it how it could be really great given time.

[nubpro]

Starting from version 2.2.0, you can now listen to DataStore events through Hub services by performing the following.
Hence, you'll be able to tell what is synced and what isn't yet.

const removeListener = Hub.listen('datastore', async (capsule) => {
      const {
        payload: { event, data },
      } = capsule;
   console.log(event, data)
});

However, this isn't documented yet. More examples here

To your first point, currently the workaround I do is I would make sure the client fully synce with the backend first. Then I would perform the query after.
In my app, I would show them a syncing screen upon starting the app, they would need to wait for everything to be synced first because they can enter the homescreen. This isn't an ideal approach I would admit, but our hands are tied.

Here's hoping that there's improvement to this.

[manueliglesias]

Hi @vlac666

Thank you for the feedback and suggestions.

It is a little frustrating to hear that the strategy has changed but the guidance for offline scenarios has not !!!

I understand why there coud be confussion regarding the apollo-based sdk and DataStore and the recomendations around the use cases and scenarios. Hopefully this helps:

The apollo solution is no longer the recomended path for offline scenarios, since it is heavily dependant on the internals of the apollo client version 2.x and it doesn't work with newer apollo client versions. Because of this, we now publish separate apollo links for auth and real time data (these two work with newer versions of apollo and let you use AppSync endpoints) but not a link for offline support.

We'll look into making it clearer in our docs.

Does a record exist or not?

As @nubpro pointed out, starting from version 2.2.0 DataStore provides visibility into its internal state, this is done using the Hub and it currently notifies multiple things, e.g.:

• The networkStatus (onnline/offline)
• The outboxStatus (the outbox holds outgoing data that hasn't been synced)
• subscriptionsEstablished (when the websocket connection and all the subscriptions have been established)
• syncQueriesStarted (when the sync for all the models starts)
• modelSynced (once for ecah model, tells you when sync for that model is done)
  
  
  
  • With this one you get information like how many records are new/updated/deleted also the type os sync (base or delta)
  
  
• syncQueriesReady (when all models have been synced)
• ready (when the overall sync process is done and all data is available locally)

I'll be sending a PR to our docs site to make this information available

My honest opinion is that it takes too much control and feedback away from developers.

What control do you feel you're losing?

[nubpro]

Hi @vlac666

Thank you for the feedback and suggestions.

It is a little frustrating to hear that the strategy has changed but the guidance for offline scenarios has not !!!

I understand why there coud be confussion regarding the apollo-based sdk and DataStore and the recomendations around the use cases and scenarios. Hopefully this helps:

The apollo solution is no longer the recomended path for offline scenarios, since it is heavily dependant on the internals of the apollo client version 2.x and it doesn't work with newer apollo client versions. Because of this, we now publish separate apollo links for auth and real time data (these two work with newer versions of apollo and let you use AppSync endpoints) but not a link for offline support.

We'll look into making it clearer in our docs.

Does a record exist or not?

As @nubpro pointed out, starting from version 2.2.0 DataStore provides visibility into its internal state, this is done using the Hub and it currently notifies multiple things, e.g.:

• The networkStatus (onnline/offline)
• The outboxStatus (the outbox holds outgoing data that hasn't been synced)
• subscriptionsEstablished (when the websocket connection and all the subscriptions have been established)
• syncQueriesStarted (when the sync for all the models starts)

• modelSynced (once for ecah model, tells you when sync for that model is done)

  • With this one you get information like how many records are new/updated/deleted also the type os sync (base or delta)
• syncQueriesReady (when all models have been synced)
• ready (when the overall sync process is done and all data is available locally)

I'll be sending a PR to our docs site to make this information available

My honest opinion is that it takes too much control and feedback away from developers.

What control do you feel you're losing?

1. Please expose the websocket status to us and also provide a method to manually reconnect the websocket without restarting DataStore. #6162
2. On another note, could you also indicate whether the data we retrieved from Datastore.observe(Model)is local or remote?
   This is going to be exceptionally handy because sometimes Hub.listener might not be around when you re-open the app with leftover mutations in the outbox.
3. Is it also possible to indicate a merge between the same record has been performed? Because I expect 1 outboxMutationEnqueued will definitely return a outboxMutationProcessed. But it doesn't because a merge has happen when the second mutation is enqueued.

[vlac666]

I understand why there coud be confussion regarding the apollo-based sdk and DataStore and the recomendations around the use cases and scenarios. Hopefully this helps:

The apollo solution is no longer the recommended path for offline scenarios, since it is heavily dependant on the internals of the apollo client version 2.x and it doesn't work with newer apollo client versions. Because of this, we now publish separate apollo links for auth and real time data (these two work with newer versions of apollo and let you use AppSync endpoints) but not a link for offline support.

We'll look into making it clearer in our docs.

Thanks for the helpful response. Much appreciated. If it helps, I think this is main page that needs updating to reflect the latest guidance API (GRAPHQL) Offline scenarios - js

[manueliglesias]

Thanks @nubpro

On another note, could you also indicate whether the data we retrieved from Datastore.observe(Model)is local or remote?

This sounds good, we could add a new key on the subscription message, kind of like this:

export type SubscriptionMessage<T extends PersistentModel> = {
    opType: OpType;
    element: T;
    model: PersistentModelConstructor<T>;
    condition: PredicatesGroup<T> | null;
+       source: 'LOCAL' | 'REMOTE'
};

For now you can look for the presence of _version, that would only help if the item is new, though.

Is it also possible to indicate a merge between the same record has been performed? Because I expect 1 outboxMutationEnqueued will definitely return a outboxMutationProcessed. But it doesn't because a merge has happen when the second mutation is enqueued.

Yeah, this sounds useful. Would a new event on the Hub indicating this help you?

[nubpro]

Thanks @nubpro

On another note, could you also indicate whether the data we retrieved from Datastore.observe(Model)is local or remote?

This sounds good, we could add a new key on the subscription message, kind of like this:

export type SubscriptionMessage<T extends PersistentModel> = {
  opType: OpType;
  element: T;
  model: PersistentModelConstructor<T>;
  condition: PredicatesGroup<T> | null;
+       source: 'LOCAL' | 'REMOTE'
};

For now you can look for the presence of _version, that would only help if the item is new, though.

Is it also possible to indicate a merge between the same record has been performed? Because I expect 1 outboxMutationEnqueued will definitely return a outboxMutationProcessed. But it doesn't because a merge has happen when the second mutation is enqueued.

Yeah, this sounds useful. Would a new event on the Hub indicating this help you?

I guess that would work too.
It turns out, exposing the remaining mutations in the outbox is much more useful for our usecase as we would like backup the mutations in one place if the user were to logout immediately before the outbox is fully processed.

How does this model merge work?
What strategy does it uses? Auto merge?