Amplify-js: DataStore would not attempt to re-establish subscriptions when subscription timeout or handshake error occured

Created on 4 Aug 2020  Â·  24Comments  Â·  Source: aws-amplify/amplify-js

Describe the bug
Subscriptions with PubSub in DataStore can be very fragile and would not re-attempt to re-estalish itself when an internet disconnection happen over a short period of time. Network drops are common and therefore DataStore's subscriptions needs the resilience to recover from such scenario.
After the merge of PR #6366, DataStore would only re-estalish the subscriptions in the case of Connection closed and Timeout disconnect. However, if there is a failure when conducting a handshake or when the client didnt receive start_ack from server, DataStore would not do anything. Do that note that, on React Native, NetInfo is unable to detect a transient network drop has occured. Hence, it will tell DataStore that the client is online throughout despite a network drop.

I have highlighted the failure point in red which DataStore subscriptions does not recover.
image

In purple, currently the client will only know that the websocket has broken off after 5 minutes when it did not receive KA from the server. In my opinion, 5 minutes is too long for PubSub to detect Timeout disconnect has happened. We would lose all the data that happened during this period, it simply doesn't cut for our project. Perhaps we can lower this down to a minute and half?

To Reproduce
Steps to reproduce the behavior:

  1. iPad > Developer setting > Network Link Conditioner > Enable Bad network.
  2. Leave the app open for few hours and check back every 1 hour to see whether the subscription remains alive or not.

This can be tricky to reproduce therefore you need to have the patience.

Expected behavior
DataStore subscriptions should recover or at least re-estalish itself when handshake error or subscription timeout has occured.
**I've made some changes to force DataStore to re-establish the subscriptions when subscription timeout has occured. However, if we would constantly re-establish the subsciptions on the same websocket over and over within a short period, the client would easily hit Max subscriptions error. I don't know what else I can do apart from this.

What is Configured?


Environment

  System:
    OS: macOS 10.15.5
    CPU: (4) x64 Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
    Memory: 179.66 MB / 8.00 GB
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 14.3.0 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.4 - /usr/local/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  Browsers:
    Chrome: 84.0.4147.105
    Safari: 13.1.1
  npmGlobalPackages:
    @aws-amplify/cli: 4.22.0
    ios-deploy: 1.10.0
    npm: 6.14.4
    react-native-cli: 2.0.1

Smartphone (please complete the following information):

  • Device: iPad
  • OS: iOS 13
  • React Native
Can't reproduce DataStore
Source
picture nubpro

Most helpful comment

I am still experiencing this due to my wifi connection intermittently dropping and reconnecting.
Data does not seem to be syncing to the cloud either, but it does when i have an uninterrupted connection.

Unhandled Rejection (Error): Connection handshake error
WebSocket.newSocket.onclose

C:/.../Providers/AWSAppSyncRealTimeProvider.ts:646
  643 | _b.trys.push([1, 4, , 5]);
  644 | return [4 /*yield*/, (function () {
  645 |         return new Promise(function (res, rej) {
> 646 |             var newSocket = new WebSocket(awsRealTimeUrl, 'graphql-ws');
      | ^  647 |             newSocket.onerror = function () {
  648 |                 logger.debug("WebSocket connection error");
  649 |             };
picture laclance on 27 Oct 2020
👍2

All 24 comments

Hi @nubpro - Just discussed with the team. First, can you to validate that this is a still an issue with our recent changes? Second, we need to check internally on reducing or customizing this timeout window.

Also, regarding your suggestion, after the reconnection happens on the client a sync process starts, so this should ensure that no data is lost. Can you confirm that this is the case?

mauerbac picture mauerbac on 20 Aug 2020

Hi @nubpro - Just discussed with the team. First, can you to validate that this is a still an issue with our recent changes? Second, we need to check internally on reducing or customizing this timeout window.

Also, regarding your suggestion, after the reconnection happens on the client a sync process starts, so this should ensure that no data is lost. Can you confirm that this is the case?

Yes, I believe this issue is still present on the latest Amplify.
I don't think there's any changes that attempt to fix this underlying issue yet.

Currently DataStore would only restart the sync process if:

  1. NetInfo detects the Wi-Fi is unreachable (by default, it would ping Google every 60s to check the network connectivity). If the device internet went offline and online, before NetInfo is able to detect. DataStore doesn't restart until Subscription throws Timeout disconnect.
  2. Subscription throws Timeout disconnect and Connection closed.

DataStore doesn't restart for the following case:

  1. Subscription timeout. This happens when DataStore re-attempts to establish the subscription after a network reconnection. But due to a network drop when the subscription is being established, Subscription will get timeout.
  2. Failed to estalish websocket due to handshake failure.

We should all understand that connectivity drops are common occurrences and currently, DataStore is simply too fragile to recover from these scenarios.

I would also want to highlight that by constantly restarting the subscriptions, we are hitting Max subscriptions error, most likely AppSync didn't know that the subscription is getting closed. You can reproduce this easily by logging in and out a few times. I'll probably create a separate ticket on this.

I hope this answered some of your question. To get a respond after 17 days is a bit disappointing honestly.

nubpro picture nubpro on 21 Aug 2020

@nubpro

Are you still experiencing this issue, I did the same thing you mentioned (iPad > Developer setting > Network Link Conditioner > Enable Bad network.) and everything works as expected.

Have you tried the latest release?

elorzafe picture elorzafe on 28 Aug 2020

@nubpro

Are you still experiencing this issue, I did the same thing you mentioned (iPad > Developer setting > Network Link Conditioner > Enable Bad network.) and everything works as expected.

Have you tried the latest release?

This is fairly difficult to reproduce but it does happen. I don't have any more details to share as we departing from using the websockets.
Hence, I highlighted the two sections where DataStore wouldn't reconnect if the scenarios happen.

nubpro picture nubpro on 28 Aug 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] picture stale[bot] on 28 Sep 2020

This issue has been automatically closed because of inactivity. Please open a new issue if are still encountering problems.

stale[bot] picture stale[bot] on 10 Oct 2020

I am still experiencing this due to my wifi connection intermittently dropping and reconnecting.
Data does not seem to be syncing to the cloud either, but it does when i have an uninterrupted connection.

Unhandled Rejection (Error): Connection handshake error
WebSocket.newSocket.onclose

C:/.../Providers/AWSAppSyncRealTimeProvider.ts:646
  643 | _b.trys.push([1, 4, , 5]);
  644 | return [4 /*yield*/, (function () {
  645 |         return new Promise(function (res, rej) {
> 646 |             var newSocket = new WebSocket(awsRealTimeUrl, 'graphql-ws');
      | ^  647 |             newSocket.onerror = function () {
  648 |                 logger.debug("WebSocket connection error");
  649 |             };
laclance picture laclance on 27 Oct 2020
👍2

@elorzafe Can we please have this issue reopened?

@laclance finally, someone has the problem as I'm having đź‘Ť

nubpro picture nubpro on 27 Oct 2020

It's confusing because one of the main reasons for datastore is online/offline syncing but then as soon as I am offline I get this error.

laclance picture laclance on 2 Nov 2020

@laclance Are you able to consistently get this behavior to happen, and if so, would you be able to share some steps? We are still having issues trying to reproduce the error.

Also, if you could share any logs ( window.LOG_LEVEL = 'DEBUG' ), that would be very helpful too.

amhinson picture amhinson on 11 Nov 2020

@laclance Are you able to consistently get this behavior to happen, and if so, would you be able to share some steps? We are still having issues trying to reproduce the error.

Also, if you could share any logs ( window.LOG_LEVEL = 'DEBUG' ), that would be very helpful too.

This is relatively difficult to reproduce as it regards to network stability.

Would it be plausible to force the problem by commenting these chunk of code?

nubpro picture nubpro on 11 Nov 2020

@nubpro Good thought. Testing locally, that does seem to throw this error consistently, but it doesn't necessarily give me much confidence with how an the library would actually handle this scenario.

With that said, enabling logging does show that PubSub is attempting to reconnect multiple times on connection timeout since jitteredExponentialRetry is used here.

I'd be interested to see what the logs look like without commenting out any code. If you could try to catch the error when attempting to reproduce, that'd be great!


Logs

ConsoleLogger.ts:91 [DEBUG] 22:06.946 AWSAppSyncRealTimeProvider - Initializaling retryable Handshake
ConsoleLogger.ts:91 [DEBUG] 22:06.947 AWSAppSyncRealTimeProvider - Initializing handshake wss://*****.appsync-realtime-api.us-west-2.amazonaws.com/graphql?header=*****
ConsoleLogger.ts:91 [DEBUG] 22:07.132 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"connection_ack","payload":{"connectionTimeoutMs":300000}} 
ConsoleLogger.ts:91 [DEBUG] 22:07.132 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:99 [DEBUG] 22:22.438 Util - error on bound  Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealTimeProvider.checkAckOk (AWSAppSyncRealTimeProvider.ts:714)
ConsoleLogger.ts:91 [DEBUG] 22:22.696 AWSAppSyncRealTimeProvider - Initializing handshake wss://*****.appsync-realtime-api.us-west-2.amazonaws.com/graphql?header=*****
ConsoleLogger.ts:91 [DEBUG] 22:22.871 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"connection_ack","payload":{"connectionTimeoutMs":300000}} 
ConsoleLogger.ts:91 [DEBUG] 22:22.871 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:99 [DEBUG] 22:38.438 Util - error on bound  Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealTimeProvider.checkAckOk (AWSAppSyncRealTimeProvider.ts:714)
ConsoleLogger.ts:91 [DEBUG] 22:39.1 AWSAppSyncRealTimeProvider - Initializing handshake wss://*****.appsync-realtime-api.us-west-2.amazonaws.com/graphql?header=*****
ConsoleLogger.ts:91 [DEBUG] 22:39.225 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"connection_ack","payload":{"connectionTimeoutMs":300000}} 
ConsoleLogger.ts:91 [DEBUG] 22:39.225 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:99 [DEBUG] 22:54.931 Util - error on bound  Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealTimeProvider.checkAckOk (AWSAppSyncRealTimeProvider.ts:714)
ConsoleLogger.ts:91 [DEBUG] 22:56.418 AWSAppSyncRealTimeProvider - Initializing handshake wss://*****.appsync-realtime-api.us-west-2.amazonaws.com/graphql?header=*****
ConsoleLogger.ts:91 [DEBUG] 22:56.612 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"connection_ack","payload":{"connectionTimeoutMs":300000}} 
ConsoleLogger.ts:91 [DEBUG] 22:56.613 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:91 [DEBUG] 23:07.107 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:99 [DEBUG] 23:11.597 Util - error on bound  Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealTimeProvider.checkAckOk (AWSAppSyncRealTimeProvider.ts:714)
ConsoleLogger.ts:91 [DEBUG] 23:13.265 AWSAppSyncRealTimeProvider - Initializing handshake wss://*****.appsync-realtime-api.us-west-2.amazonaws.com/graphql?header=*****
ConsoleLogger.ts:91 [DEBUG] 23:13.646 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"connection_ack","payload":{"connectionTimeoutMs":300000}} 
ConsoleLogger.ts:91 [DEBUG] 23:13.647 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:91 [DEBUG] 23:22.843 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:99 [DEBUG] 23:28.929 Util - error on bound  Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealTimeProvider.checkAckOk (AWSAppSyncRealTimeProvider.ts:714)
ConsoleLogger.ts:91 [DEBUG] 23:33.709 AWSAppSyncRealTimeProvider - Initializing handshake wss://*****.appsync-realtime-api.us-west-2.amazonaws.com/graphql?header=*****
ConsoleLogger.ts:91 [DEBUG] 23:34.8 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"connection_ack","payload":{"connectionTimeoutMs":300000}} 
ConsoleLogger.ts:91 [DEBUG] 23:34.9 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:91 [DEBUG] 23:39.215 AWSAppSyncRealTimeProvider - subscription message from AWS AppSyncRealTime: {"type":"ka"} 
ConsoleLogger.ts:99 [DEBUG] 23:48.990 Util - error on bound  Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealTimeProvider.checkAckOk (AWSAppSyncRealTimeProvider.ts:714)
ConsoleLogger.ts:93 [DEBUG] 23:48.990 AWSAppSyncRealTimeProvider {err: Error: Connection timeout: ack from AWSRealTime was not received on 15000 ms
    at AWSAppSyncRealT…}
react_devtools_backend.js:2450 [WARN] 23:49.28 DataStore - subscriptionError Connection failed: Connection timeout: ack from AWSRealTime was not received on 15000 ms

amhinson picture amhinson on 11 Nov 2020

yes I just have to switch off my internet to get the error. 

[DEBUG] 28:22.241 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
AWSAppSyncRealTimeProvider.ts:646 Uncaught (in promise) Error: Connection handshake error
    at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
laclance picture laclance on 12 Nov 2020

@laclance Could you post for of the logging output? I'm mainly interested in what happens leading up to when the error occurs. And be sure to remove any sensitive data that might be contained in the logs.

amhinson picture amhinson on 12 Nov 2020


Logs

[DEBUG] 39:05.22 DataStore - GraphQL endpoint available https://###.appsync-api.eu-west-1.amazonaws.com/graphql
  ConsoleLogger.ts:91 [INFO] 39:05.38 DataStore - starting sync engine...
ConsoleLogger.ts:91 [DEBUG] 39:05.438 AuthClass - getting current credentials
ConsoleLogger.ts:91 [DEBUG] 39:05.439 Credentials - getting credentials
ConsoleLogger.ts:91 [DEBUG] 39:05.440 Credentials - picking up credentials
ConsoleLogger.ts:91 [DEBUG] 39:05.440 Credentials - getting new cred promise
ConsoleLogger.ts:91 [DEBUG] 39:05.441 Credentials - checking if credentials exists and not expired
ConsoleLogger.ts:91 [DEBUG] 39:05.442 Credentials - need to get a new credential or refresh the existing one
ConsoleLogger.ts:91 [DEBUG] 39:05.444 AuthClass - Getting current user credentials
ConsoleLogger.ts:91 [DEBUG] 39:05.447 AuthClass - Getting current session
ConsoleLogger.ts:91 [DEBUG] 39:05.450 AuthClass - Failed to get user from user pool
ConsoleLogger.ts:99 [DEBUG] 39:05.452 AuthClass - Failed to get the current user No current user
ConsoleLogger.ts:99 [DEBUG] 39:05.452 AuthClass - getting session failed No current user
ConsoleLogger.ts:91 [DEBUG] 39:05.453 Credentials - setting credentials for guest
  VM26:1 POST https://cognito-identity.eu-west-1.amazonaws.com/ net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ VM26:1
FetchHttpHandler.handle @ fetch-http-handler.ts:61
(anonymous) @ GetIdCommand.ts:48
(anonymous) @ deserializerMiddleware.ts:16
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
(anonymous) @ deserializerMiddleware.ts:14
(anonymous) @ loggerMiddleware.ts:22
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
(anonymous) @ loggerMiddleware.ts:18
(anonymous) @ defaultStrategy.ts:118
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
step @ tslib.es6.js:85
(anonymous) @ tslib.es6.js:81
fulfilled @ tslib.es6.js:71
Promise.then (async)
step @ tslib.es6.js:73
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
StandardRetryStrategy.retry @ defaultStrategy.ts:98
(anonymous) @ retryMiddleware.ts:17
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
(anonymous) @ retryMiddleware.ts:16
(anonymous) @ index.ts:32
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
(anonymous) @ index.ts:19
(anonymous) @ index.ts:38
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
(anonymous) @ index.ts:17
(anonymous) @ middleware.ts:29
(anonymous) @ serializerMiddleware.ts:18
step @ tslib.es6.js:100
(anonymous) @ tslib.es6.js:81
fulfilled @ tslib.es6.js:71
Promise.then (async)
step @ tslib.es6.js:73
(anonymous) @ tslib.es6.js:74
__awaiter @ tslib.es6.js:70
(anonymous) @ serializerMiddleware.ts:15
Client.send @ client.ts:55
(anonymous) @ Credentials.ts:266
step @ index.ts:29
(anonymous) @ index.ts:29
(anonymous) @ index.ts:29
push../node_modules/@aws-amplify/core/lib-esm/Credentials.js.__awaiter @ index.ts:29
credentialsProvider @ Credentials.ts:265
(anonymous) @ Credentials.ts:284
step @ index.ts:29
(anonymous) @ index.ts:29
fulfilled @ index.ts:29
Promise.then (async)
step @ index.ts:29
(anonymous) @ index.ts:29
push../node_modules/@aws-amplify/core/lib-esm/Credentials.js.__awaiter @ index.ts:29
CredentialsClass._setCredentialsForGuest @ Credentials.ts:212
CredentialsClass.set @ Credentials.ts:517
(anonymous) @ Auth.ts:1388
Promise.catch (async)
(anonymous) @ Auth.ts:1385
step @ Auth.ts:12
(anonymous) @ Auth.ts:12
fulfilled @ Auth.ts:12
Promise.then (async)
step @ Auth.ts:12
(anonymous) @ Auth.ts:12
push../node_modules/@aws-amplify/auth/lib-esm/Auth.js.__awaiter @ Auth.ts:12
AuthClass.currentUserCredentials @ Auth.ts:1357
CredentialsClass._keepAlive @ Credentials.ts:112
CredentialsClass._pickupCredentials @ Credentials.ts:90
CredentialsClass.get @ Credentials.ts:83
AuthClass.currentCredentials @ Auth.ts:1395
(anonymous) @ subscription.ts:248
step @ mutation.ts:442
(anonymous) @ mutation.ts:442
(anonymous) @ mutation.ts:442
push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
(anonymous) @ subscription.ts:244
(anonymous) @ subscription.ts:428
Subscription @ Observable.js:197
subscribe @ Observable.js:279
(anonymous) @ index.ts:682
(anonymous) @ index.ts:681
step @ index.ts:3
(anonymous) @ index.ts:3
(anonymous) @ index.ts:3
push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
SyncEngine.waitForSubscriptionsReady @ index.ts:677
(anonymous) @ index.ts:198
step @ index.ts:3
(anonymous) @ index.ts:3
(anonymous) @ index.ts:3
push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
(anonymous) @ index.ts:162
notifySubscription @ Observable.js:135
onNotify @ Observable.js:179
next @ Observable.js:235
(anonymous) @ datastoreConnectivity.ts:36
notifySubscription @ Observable.js:135
flushSubscription @ Observable.js:121
(anonymous) @ Observable.js:174
(anonymous) @ Observable.js:73
Promise.then (async)
enqueue @ Observable.js:71
onNotify @ Observable.js:173
next @ Observable.js:235
(anonymous) @ Reachability.ts:21
Subscription @ Observable.js:197
subscribe @ Observable.js:279
(anonymous) @ datastoreConnectivity.ts:31
Subscription @ Observable.js:197
subscribe @ Observable.js:279
(anonymous) @ index.ts:162
(anonymous) @ index.ts:161
step @ index.ts:3
(anonymous) @ index.ts:3
fulfilled @ index.ts:3
Promise.then (async)
step @ index.ts:3
(anonymous) @ index.ts:3
push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
(anonymous) @ index.ts:151
(anonymous) @ index.ts:400
Subscription @ Observable.js:197
subscribe @ Observable.js:279
(anonymous) @ datastore.ts:607
step @ jsSha256.ts:110
(anonymous) @ jsSha256.ts:110
fulfilled @ jsSha256.ts:110
Promise.then (async)
step @ jsSha256.ts:110
fulfilled @ jsSha256.ts:110
Promise.then (async)
step @ jsSha256.ts:110
fulfilled @ jsSha256.ts:110
Promise.then (async)
step @ jsSha256.ts:110
(anonymous) @ jsSha256.ts:110
push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
DataStore.start @ datastore.ts:559
(anonymous) @ datastore.ts:967
step @ jsSha256.ts:110
(anonymous) @ jsSha256.ts:110
(anonymous) @ jsSha256.ts:110
push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
(anonymous) @ datastore.ts:966
(anonymous) @ datastore.ts:973
Subscription @ Observable.js:197
subscribe @ Observable.js:279
(anonymous) @ MerchantContextProvider.tsx:143
invokePassiveEffectCreate @ react-dom.development.js:23487
callCallback @ react-dom.development.js:3945
invokeGuardedCallbackDev @ react-dom.development.js:3994
invokeGuardedCallback @ react-dom.development.js:4056
flushPassiveEffectsImpl @ react-dom.development.js:23574
unstable_runWithPriority @ scheduler.development.js:646
runWithPriority$1 @ react-dom.development.js:11276
flushPassiveEffects @ react-dom.development.js:23447
performSyncWorkOnRoot @ react-dom.development.js:22269
(anonymous) @ react-dom.development.js:11327
unstable_runWithPriority @ scheduler.development.js:646
runWithPriority$1 @ react-dom.development.js:11276
flushSyncCallbackQueueImpl @ react-dom.development.js:11322
flushSyncCallbackQueue @ react-dom.development.js:11309
scheduleUpdateOnFiber @ react-dom.development.js:21893
enqueueSetState @ react-dom.development.js:12467
push../node_modules/react/cjs/react.development.js.Component.setState @ react.development.js:365
(anonymous) @ react.js:61
dispatch @ redux.js:222
rehydrate @ persistStore.js:79
_rehydrate @ persistReducer.js:61
(anonymous) @ persistReducer.js:90
Promise.then (async)
(anonymous) @ persistReducer.js:89
Promise.then (async)
(anonymous) @ persistReducer.js:84
p @ VM31:1
v @ VM31:1
(anonymous) @ VM31:1
dispatch @ redux.js:213
e @ VM31:1
(anonymous) @ redux-logger.js:1
dispatch @ VM31:1
persist @ persistStore.js:114
persistStore @ persistStore.js:123
push../src/redux/configureStore.ts.__webpack_exports__.default @ configureStore.ts:15
  ./src/index.tsx @ index.tsx:21
__webpack_require__ @ bootstrap:784
fn @ bootstrap:150
1 @ utils.ts:40
__webpack_require__ @ bootstrap:784
checkDeferredModules @ bootstrap:45
webpackJsonpCallback @ bootstrap:32
(anonymous) @ main.chunk.js:1
Show 74 more frames
ConsoleLogger.ts:99 [DEBUG] 39:05.483 Credentials - Failed to load credentials Promise {<rejected>: TypeError: Failed to fetch}
  ConsoleLogger.ts:99 [DEBUG] 39:05.484 Credentials - Error loading credentials TypeError: Failed to fetch
  ConsoleLogger.ts:91 [DEBUG] 39:05.484 AuthClass - Getting current session
  ConsoleLogger.ts:91 [DEBUG] 39:05.485 AuthClass - Failed to get user from user pool
  ConsoleLogger.ts:99 [DEBUG] 39:05.485 AuthClass - Failed to get the current user No current user
  ConsoleLogger.ts:91 [INFO] 39:05.486 Cache - Get item: key is federatedInfo with options undefined
  ConsoleLogger.ts:91 [DEBUG] 39:05.487 AuthClass - getting current authenticated user
  ConsoleLogger.ts:91 [DEBUG] 39:05.487 AuthClass - get current authenticated userpool user
  ConsoleLogger.ts:91 [DEBUG] 39:05.488 AuthClass - Failed to get user from user pool
  ConsoleLogger.ts:99 [DEBUG] 39:05.488 AuthClass - The user is not authenticated by the error No current user
  ConsoleLogger.ts:99 [DEBUG] 39:05.488 DataStore - error getting OIDC JWT not authenticated
  ConsoleLogger.ts:99 [DEBUG] 39:05.495 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateUser {↵    id↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.502 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateUser {↵    id↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.505 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteUser {↵    id↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.508 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateStaff {↵    id↵…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.510 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateStaff {↵    id↵…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.512 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteStaff {↵    id↵…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.515 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateShift {↵    id↵…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.518 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateShift {↵    id↵…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.521 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteShift {↵    id↵…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.523 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateTab {↵    id↵  …  customer {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.529 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateTab {↵    id↵  …  customer {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.545 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteTab {↵    id↵  …  customer {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.550 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateOrderLine {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.552 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateOrderLine {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.555 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteOrderLine {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.557 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateTabPayment {↵  …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.558 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateTabPayment {↵  …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.560 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteTabPayment {↵  …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.562 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreatePaymentType {↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.565 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdatePaymentType {↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.567 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeletePaymentType {↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.570 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateTabAudit {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.571 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateTabAudit {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.572 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteTabAudit {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.574 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateCustomer {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.575 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateCustomer {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.576 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteCustomer {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.578 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateSellingLocation…  merchant {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.579 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateSellingLocation…  merchant {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.580 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteSellingLocation…  merchant {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.582 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateMerchant {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.584 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateMerchant {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.585 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteMerchant {↵    …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.587 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreatePOSDevice {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.589 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdatePOSDevice {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.591 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeletePOSDevice {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.594 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateProduct {↵    i…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.601 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateProduct {↵    i…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.603 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteProduct {↵    i…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.604 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateProductPack {↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.605 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateProductPack {↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.606 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteProductPack {↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.608 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateProductPackPric…roductPack {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.609 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateProductPackPric…roductPack {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.610 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteProductPackPric…roductPack {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.611 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreatePriceList {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.613 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdatePriceList {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.614 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeletePriceList {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.615 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateProductModifier…ifierGroup {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.616 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateProductModifier…ifierGroup {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.618 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteProductModifier…ifierGroup {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.621 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateModifierGroup {…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.623 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateModifierGroup {…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.625 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteModifierGroup {…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.627 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateProductModifier…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.630 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateProductModifier…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.632 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteProductModifier…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.634 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateSellingLocation…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.636 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateSellingLocation…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.639 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteSellingLocation…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.643 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateSetting {↵    i…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.645 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateSetting {↵    i…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.646 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteSetting {↵    i…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.649 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateSellingLocation…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.652 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateSellingLocation…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.654 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteSellingLocation…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.657 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateMenu {↵    id↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.659 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateMenu {↵    id↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.660 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteMenu {↵    id↵ …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.661 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateCategory {↵    …ntCategory {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.663 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateCategory {↵    …ntCategory {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.664 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteCategory {↵    …ntCategory {↵      id↵      _deleted↵    }↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.665 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateDownstreamPOSEv…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.667 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateDownstreamPOSEv…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.668 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteDownstreamPOSEv…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.669 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateUpstreamPOSEven…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.670 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateUpstreamPOSEven…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.672 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteUpstreamPOSEven…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.673 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateOrderLineModifi…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.674 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateOrderLineModifi…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.675 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteOrderLineModifi…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.677 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onCreateOrderLineProduc…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.678 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onUpdateOrderLineProduc…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:99 [DEBUG] 39:05.679 PubSub - subscribe options {provider: Symbol(INTERNAL_AWS_APPSYNC_REALTIME_PUBSUB_PROVIDER), appSyncGraphqlEndpoint: "https://###.appsync-api.eu-west-1.amazonaws.com/graphql", authenticationType: "API_KEY", apiKey: "da2-###", query: "subscription operation {↵  onDeleteOrderLineProduc…  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", …}
  ConsoleLogger.ts:91 [DEBUG] 39:05.697 AWSAppSyncRealTimeProvider - Initializaling retryable Handshake
  ConsoleLogger.ts:91 [DEBUG] 39:05.699 Util - bound  attempt #1 with this vars: [{"awsRealTimeUrl":"wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30="}]
  ConsoleLogger.ts:91 [DEBUG] 39:05.701 AWSAppSyncRealTimeProvider - Initializing handshake wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=
  AWSAppSyncRealTimeProvider.ts:641 WebSocket connection to 'wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:641
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:640
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:653
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeHandshake @ AWSAppSyncRealTimeProvider.ts:634
  (anonymous) @ Retry.ts:37
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  push../node_modules/@aws-amplify/core/lib-esm/Util/Retry.js.__awaiter @ Reachability.ts:8
  retry @ Retry.ts:26
  jitteredExponentialRetry @ Retry.ts:78
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:627
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeRetryableHandshake @ AWSAppSyncRealTimeProvider.ts:625
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:599
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  fulfilled @ AWSAppSyncProvider.ts:204
  ConsoleLogger.ts:91 [DEBUG] 39:05.726 AWSAppSyncRealTimeProvider - WebSocket connection error
  ConsoleLogger.ts:99 [DEBUG] 39:05.726 Util - error on bound  Error: Connection handshake error
  at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
  ConsoleLogger.ts:91 [DEBUG] 39:05.727 Util - bound  retrying in 216.51369021981324 ms
  ConsoleLogger.ts:91 [DEBUG] 39:05.946 Util - bound  attempt #2 with this vars: [{"awsRealTimeUrl":"wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30="}]
  ConsoleLogger.ts:91 [DEBUG] 39:05.946 AWSAppSyncRealTimeProvider - Initializing handshake wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=
  AWSAppSyncRealTimeProvider.ts:641 WebSocket connection to 'wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:641
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:640
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:653
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeHandshake @ AWSAppSyncRealTimeProvider.ts:634
  (anonymous) @ Retry.ts:37
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  push../node_modules/@aws-amplify/core/lib-esm/Util/Retry.js.__awaiter @ Reachability.ts:8
  retry @ Retry.ts:26
  (anonymous) @ Retry.ts:51
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  fulfilled @ Reachability.ts:8
  ConsoleLogger.ts:91 [DEBUG] 39:05.956 AWSAppSyncRealTimeProvider - WebSocket connection error
  ConsoleLogger.ts:99 [DEBUG] 39:05.957 Util - error on bound  Error: Connection handshake error
  at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
  ConsoleLogger.ts:91 [DEBUG] 39:05.958 Util - bound  retrying in 413.4805285872718 ms
  ConsoleLogger.ts:91 [DEBUG] 39:06.378 Util - bound  attempt #3 with this vars: [{"awsRealTimeUrl":"wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30="}]
  ConsoleLogger.ts:91 [DEBUG] 39:06.378 AWSAppSyncRealTimeProvider - Initializing handshake wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=
  AWSAppSyncRealTimeProvider.ts:641 WebSocket connection to 'wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:641
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:640
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:653
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeHandshake @ AWSAppSyncRealTimeProvider.ts:634
  (anonymous) @ Retry.ts:37
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  push../node_modules/@aws-amplify/core/lib-esm/Util/Retry.js.__awaiter @ Reachability.ts:8
  retry @ Retry.ts:26
  (anonymous) @ Retry.ts:51
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  fulfilled @ Reachability.ts:8
  ConsoleLogger.ts:91 [DEBUG] 39:06.386 AWSAppSyncRealTimeProvider - WebSocket connection error
  ConsoleLogger.ts:99 [DEBUG] 39:06.386 Util - error on bound  Error: Connection handshake error
  at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
  ConsoleLogger.ts:91 [DEBUG] 39:06.387 Util - bound  retrying in 808.955420468182 ms
  ConsoleLogger.ts:91 [DEBUG] 39:07.208 Util - bound  attempt #4 with this vars: [{"awsRealTimeUrl":"wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30="}]
  ConsoleLogger.ts:91 [DEBUG] 39:07.208 AWSAppSyncRealTimeProvider - Initializing handshake wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=
  AWSAppSyncRealTimeProvider.ts:641 WebSocket connection to 'wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:641
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:640
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:653
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeHandshake @ AWSAppSyncRealTimeProvider.ts:634
  (anonymous) @ Retry.ts:37
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  push../node_modules/@aws-amplify/core/lib-esm/Util/Retry.js.__awaiter @ Reachability.ts:8
  retry @ Retry.ts:26
  (anonymous) @ Retry.ts:51
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  fulfilled @ Reachability.ts:8
  ConsoleLogger.ts:91 [DEBUG] 39:07.217 AWSAppSyncRealTimeProvider - WebSocket connection error
  ConsoleLogger.ts:99 [DEBUG] 39:07.218 Util - error on bound  Error: Connection handshake error
  at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
  ConsoleLogger.ts:91 [DEBUG] 39:07.219 Util - bound  retrying in 1692.9802996295768 ms
  ConsoleLogger.ts:91 [DEBUG] 39:08.911 Util - bound  attempt #5 with this vars: [{"awsRealTimeUrl":"wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30="}]
  ConsoleLogger.ts:91 [DEBUG] 39:08.912 AWSAppSyncRealTimeProvider - Initializing handshake wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=
  AWSAppSyncRealTimeProvider.ts:641 WebSocket connection to 'wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:641
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:640
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:653
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeHandshake @ AWSAppSyncRealTimeProvider.ts:634
  (anonymous) @ Retry.ts:37
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  push../node_modules/@aws-amplify/core/lib-esm/Util/Retry.js.__awaiter @ Reachability.ts:8
  retry @ Retry.ts:26
  (anonymous) @ Retry.ts:51
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  fulfilled @ Reachability.ts:8
  ConsoleLogger.ts:91 [DEBUG] 39:08.919 AWSAppSyncRealTimeProvider - WebSocket connection error
  ConsoleLogger.ts:99 [DEBUG] 39:08.919 Util - error on bound  Error: Connection handshake error
  at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
  ConsoleLogger.ts:91 [DEBUG] 39:08.920 Util - bound  retrying in 3252.1393715201875 ms
  ConsoleLogger.ts:91 [DEBUG] 39:12.177 Util - bound  attempt #6 with this vars: [{"awsRealTimeUrl":"wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30="}]
  ConsoleLogger.ts:91 [DEBUG] 39:12.177 AWSAppSyncRealTimeProvider - Initializing handshake wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=
  AWSAppSyncRealTimeProvider.ts:641 WebSocket connection to 'wss://###.appsync-realtime-api.eu-west-1.amazonaws.com/graphql?header=###&payload=e30=' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:641
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:640
  (anonymous) @ AWSAppSyncRealTimeProvider.ts:653
  step @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  (anonymous) @ AWSAppSyncProvider.ts:204
  push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
  AWSAppSyncRealTimeProvider._initializeHandshake @ AWSAppSyncRealTimeProvider.ts:634
  (anonymous) @ Retry.ts:37
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  push../node_modules/@aws-amplify/core/lib-esm/Util/Retry.js.__awaiter @ Reachability.ts:8
  retry @ Retry.ts:26
  (anonymous) @ Retry.ts:51
  step @ Reachability.ts:8
  (anonymous) @ Reachability.ts:8
  fulfilled @ Reachability.ts:8
  ConsoleLogger.ts:91 [DEBUG] 39:12.186 AWSAppSyncRealTimeProvider - WebSocket connection error
  ConsoleLogger.ts:99 [DEBUG] 39:12.186 Util - error on bound  Error: Connection handshake error
  at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
  ConsoleLogger.ts:91 [DEBUG] 39:12.186 Util - bound  retrying in false ms
  ConsoleLogger.ts:93 [DEBUG] 39:12.188 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    react_devtools_backend.js:2430 [WARN] 39:12.190 DataStore - subscriptionError Connection failed: Connection handshake error
    overrideMethod @ react_devtools_backend.js:2430
    ConsoleLogger._log @ ConsoleLogger.ts:99
    ConsoleLogger.warn @ ConsoleLogger.ts:132
    error @ subscription.ts:393
    notifySubscription @ Observable.js:140
    onNotify @ Observable.js:179
    error @ Observable.js:240
    error @ Observable.js:330
    notifySubscription @ Observable.js:140
    onNotify @ Observable.js:179
    error @ Observable.js:240
    error @ PubSub.ts:174
    notifySubscription @ Observable.js:140
    onNotify @ Observable.js:179
    error @ Observable.js:240
    (anonymous) @ AWSAppSyncRealTimeProvider.ts:296
    step @ AWSAppSyncProvider.ts:204
    (anonymous) @ AWSAppSyncProvider.ts:204
    rejected @ AWSAppSyncProvider.ts:204
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    fulfilled @ AWSAppSyncProvider.ts:204
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    fulfilled @ AWSAppSyncProvider.ts:204
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    (anonymous) @ AWSAppSyncProvider.ts:204
    push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
    AWSAppSyncRealTimeProvider._startSubscriptionWithAWSAppSyncRealTime @ AWSAppSyncRealTimeProvider.ts:227
    (anonymous) @ AWSAppSyncRealTimeProvider.ts:185
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ PubSub.ts:171
    (anonymous) @ PubSub.ts:170
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ Observable.js:319
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ subscription.ts:342
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
    (anonymous) @ subscription.ts:316
    (anonymous) @ subscription.ts:308
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
    (anonymous) @ subscription.ts:292
    (anonymous) @ subscription.ts:291
    (anonymous) @ subscription.ts:289
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    rejected @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    fulfilled @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    rejected @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    fulfilled @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
    (anonymous) @ subscription.ts:244
    (anonymous) @ subscription.ts:428
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ index.ts:682
    (anonymous) @ index.ts:681
    step @ index.ts:3
    (anonymous) @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    SyncEngine.waitForSubscriptionsReady @ index.ts:677
    (anonymous) @ index.ts:198
    step @ index.ts:3
    (anonymous) @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    (anonymous) @ index.ts:162
    notifySubscription @ Observable.js:135
    onNotify @ Observable.js:179
    next @ Observable.js:235
    (anonymous) @ datastoreConnectivity.ts:36
    notifySubscription @ Observable.js:135
    flushSubscription @ Observable.js:121
    (anonymous) @ Observable.js:174
    (anonymous) @ Observable.js:73
    Promise.then (async)
    enqueue @ Observable.js:71
    onNotify @ Observable.js:173
    next @ Observable.js:235
    (anonymous) @ Reachability.ts:21
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ datastoreConnectivity.ts:31
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ index.ts:162
    (anonymous) @ index.ts:161
    step @ index.ts:3
    (anonymous) @ index.ts:3
    fulfilled @ index.ts:3
    Promise.then (async)
    step @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    (anonymous) @ index.ts:151
    (anonymous) @ index.ts:400
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ datastore.ts:607
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
    DataStore.start @ datastore.ts:559
    (anonymous) @ datastore.ts:967
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
    (anonymous) @ datastore.ts:966
    (anonymous) @ datastore.ts:973
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ MerchantContextProvider.tsx:143
    invokePassiveEffectCreate @ react-dom.development.js:23487
    callCallback @ react-dom.development.js:3945
    invokeGuardedCallbackDev @ react-dom.development.js:3994
    invokeGuardedCallback @ react-dom.development.js:4056
    flushPassiveEffectsImpl @ react-dom.development.js:23574
    unstable_runWithPriority @ scheduler.development.js:646
    runWithPriority$1 @ react-dom.development.js:11276
    flushPassiveEffects @ react-dom.development.js:23447
    performSyncWorkOnRoot @ react-dom.development.js:22269
    (anonymous) @ react-dom.development.js:11327
    unstable_runWithPriority @ scheduler.development.js:646
    runWithPriority$1 @ react-dom.development.js:11276
    flushSyncCallbackQueueImpl @ react-dom.development.js:11322
    flushSyncCallbackQueue @ react-dom.development.js:11309
    scheduleUpdateOnFiber @ react-dom.development.js:21893
    enqueueSetState @ react-dom.development.js:12467
    push../node_modules/react/cjs/react.development.js.Component.setState @ react.development.js:365
    (anonymous) @ react.js:61
    dispatch @ redux.js:222
    rehydrate @ persistStore.js:79
    _rehydrate @ persistReducer.js:61
    (anonymous) @ persistReducer.js:90
    Promise.then (async)
    (anonymous) @ persistReducer.js:89
    Promise.then (async)
    (anonymous) @ persistReducer.js:84
    p @ VM31:1
    v @ VM31:1
    (anonymous) @ VM31:1
    dispatch @ redux.js:213
    e @ VM31:1
    (anonymous) @ redux-logger.js:1
    dispatch @ VM31:1
    persist @ persistStore.js:114
    persistStore @ persistStore.js:123
    push../src/redux/configureStore.ts.__webpack_exports__.default @ configureStore.ts:15
    ./src/index.tsx @ index.tsx:21
    __webpack_require__ @ bootstrap:784
    fn @ bootstrap:150
    1 @ utils.ts:40
    __webpack_require__ @ bootstrap:784
    checkDeferredModules @ bootstrap:45
    webpackJsonpCallback @ bootstrap:32
    (anonymous) @ main.chunk.js:1
    Show 36 more frames
    ConsoleLogger.ts:93 [DEBUG] 39:12.218 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.219 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.219 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.220 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.220 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.221 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.221 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.222 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.222 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.223 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.223 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.224 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.224 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.225 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.225 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.226 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.226 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.227 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.228 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.229 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.229 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.230 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.230 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.231 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.231 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.232 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.232 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.233 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.234 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.234 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.235 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.235 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.236 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.237 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.238 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.238 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.239 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.240 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.241 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.241 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.242 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.244 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.246 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.247 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.248 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.249 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.250 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.250 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.251 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.252 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.253 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.253 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.254 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.254 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.255 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.255 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.256 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.256 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.257 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.258 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.259 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.260 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.261 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.262 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.262 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.263 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.264 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.265 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.266 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.267 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.267 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.268 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.268 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.269 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.269 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.270 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.270 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.271 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.271 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.272 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.272 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.273 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    ConsoleLogger.ts:93 [DEBUG] 39:12.273 AWSAppSyncRealTimeProvider {err: Error: Connection handshake error
    at WebSocket.newSocket.onclose (http://localhost:3001/static/…}
    react_devtools_backend.js:2430 [WARN] 39:12.274 DataStore - Sync error Connection failed: Connection handshake error
    overrideMethod @ react_devtools_backend.js:2430
    ConsoleLogger._log @ ConsoleLogger.ts:99
    ConsoleLogger.warn @ ConsoleLogger.ts:132
    error @ datastore.ts:626
    notifySubscription @ Observable.js:140
    onNotify @ Observable.js:179
    error @ Observable.js:240
    (anonymous) @ index.ts:204
    step @ index.ts:3
    (anonymous) @ index.ts:3
    rejected @ index.ts:3
    Promise.then (async)
    step @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    (anonymous) @ index.ts:162
    notifySubscription @ Observable.js:135
    onNotify @ Observable.js:179
    next @ Observable.js:235
    (anonymous) @ datastoreConnectivity.ts:36
    notifySubscription @ Observable.js:135
    flushSubscription @ Observable.js:121
    (anonymous) @ Observable.js:174
    (anonymous) @ Observable.js:73
    Promise.then (async)
    enqueue @ Observable.js:71
    onNotify @ Observable.js:173
    next @ Observable.js:235
    (anonymous) @ Reachability.ts:21
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ datastoreConnectivity.ts:31
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ index.ts:162
    (anonymous) @ index.ts:161
    step @ index.ts:3
    (anonymous) @ index.ts:3
    fulfilled @ index.ts:3
    Promise.then (async)
    step @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    (anonymous) @ index.ts:151
    (anonymous) @ index.ts:400
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ datastore.ts:607
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
    DataStore.start @ datastore.ts:559
    (anonymous) @ datastore.ts:967
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
    (anonymous) @ datastore.ts:966
    (anonymous) @ datastore.ts:973
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ MerchantContextProvider.tsx:143
    invokePassiveEffectCreate @ react-dom.development.js:23487
    callCallback @ react-dom.development.js:3945
    invokeGuardedCallbackDev @ react-dom.development.js:3994
    invokeGuardedCallback @ react-dom.development.js:4056
    flushPassiveEffectsImpl @ react-dom.development.js:23574
    unstable_runWithPriority @ scheduler.development.js:646
    runWithPriority$1 @ react-dom.development.js:11276
    flushPassiveEffects @ react-dom.development.js:23447
    performSyncWorkOnRoot @ react-dom.development.js:22269
    (anonymous) @ react-dom.development.js:11327
    unstable_runWithPriority @ scheduler.development.js:646
    runWithPriority$1 @ react-dom.development.js:11276
    flushSyncCallbackQueueImpl @ react-dom.development.js:11322
    flushSyncCallbackQueue @ react-dom.development.js:11309
    scheduleUpdateOnFiber @ react-dom.development.js:21893
    enqueueSetState @ react-dom.development.js:12467
    push../node_modules/react/cjs/react.development.js.Component.setState @ react.development.js:365
    (anonymous) @ react.js:61
    dispatch @ redux.js:222
    rehydrate @ persistStore.js:79
    _rehydrate @ persistReducer.js:61
    (anonymous) @ persistReducer.js:90
    Promise.then (async)
    (anonymous) @ persistReducer.js:89
    Promise.then (async)
    (anonymous) @ persistReducer.js:84
    p @ VM31:1
    v @ VM31:1
    (anonymous) @ VM31:1
    dispatch @ redux.js:213
    e @ VM31:1
    (anonymous) @ redux-logger.js:1
    dispatch @ VM31:1
    persist @ persistStore.js:114
    persistStore @ persistStore.js:123
    push../src/redux/configureStore.ts.__webpack_exports__.default @ configureStore.ts:15
    ./src/index.tsx @ index.tsx:21
    __webpack_require__ @ bootstrap:784
    fn @ bootstrap:150
    1 @ utils.ts:40
    __webpack_require__ @ bootstrap:784
    checkDeferredModules @ bootstrap:45
    webpackJsonpCallback @ bootstrap:32
    (anonymous) @ main.chunk.js:1
    Show 36 more frames
    ConsoleLogger.ts:91 [DEBUG] 39:12.277 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.278 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.279 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.280 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.281 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.282 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.283 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.284 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.285 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.286 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.287 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.288 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.289 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    4ConsoleLogger.ts:91 [DEBUG] 39:12.290 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.291 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.292 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.293 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.294 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.295 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.296 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.297 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.298 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.299 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.300 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    5ConsoleLogger.ts:91 [DEBUG] 39:12.301 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    3ConsoleLogger.ts:91 [DEBUG] 39:12.302 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    4ConsoleLogger.ts:91 [DEBUG] 39:12.303 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    4ConsoleLogger.ts:91 [DEBUG] 39:12.304 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    4ConsoleLogger.ts:91 [DEBUG] 39:12.305 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    4ConsoleLogger.ts:91 [DEBUG] 39:12.306 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    2ConsoleLogger.ts:91 [DEBUG] 39:12.307 AWSAppSyncRealTimeProvider - Error while unsubscribing undefined
    AWSAppSyncRealTimeProvider.ts:646 Uncaught (in promise) Error: Connection handshake error
    at WebSocket.newSocket.onclose (AWSAppSyncRealTimeProvider.ts:646)
    newSocket.onclose @ AWSAppSyncRealTimeProvider.ts:646
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    fulfilled @ AWSAppSyncProvider.ts:204
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    (anonymous) @ AWSAppSyncProvider.ts:204
    push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
    (anonymous) @ AWSAppSyncRealTimeProvider.ts:569
    AWSAppSyncRealTimeProvider._initializeWebSocketConnection @ AWSAppSyncRealTimeProvider.ts:569
    (anonymous) @ AWSAppSyncRealTimeProvider.ts:287
    step @ AWSAppSyncProvider.ts:204
    (anonymous) @ AWSAppSyncProvider.ts:204
    fulfilled @ AWSAppSyncProvider.ts:204
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    fulfilled @ AWSAppSyncProvider.ts:204
    Promise.then (async)
    step @ AWSAppSyncProvider.ts:204
    (anonymous) @ AWSAppSyncProvider.ts:204
    push../node_modules/@aws-amplify/pubsub/lib-esm/Providers/AWSAppSyncRealTimeProvider.js.__awaiter @ AWSAppSyncProvider.ts:204
    AWSAppSyncRealTimeProvider._startSubscriptionWithAWSAppSyncRealTime @ AWSAppSyncRealTimeProvider.ts:227
    (anonymous) @ AWSAppSyncRealTimeProvider.ts:185
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ PubSub.ts:171
    (anonymous) @ PubSub.ts:170
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ Observable.js:319
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ subscription.ts:342
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
    (anonymous) @ subscription.ts:316
    (anonymous) @ subscription.ts:308
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
    (anonymous) @ subscription.ts:292
    (anonymous) @ subscription.ts:291
    (anonymous) @ subscription.ts:289
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    rejected @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    fulfilled @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    rejected @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    fulfilled @ mutation.ts:442
    Promise.then (async)
    step @ mutation.ts:442
    (anonymous) @ mutation.ts:442
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/processors/subscription.js.__awaiter @ mutation.ts:442
    (anonymous) @ subscription.ts:244
    (anonymous) @ subscription.ts:428
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ index.ts:682
    (anonymous) @ index.ts:681
    step @ index.ts:3
    (anonymous) @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    SyncEngine.waitForSubscriptionsReady @ index.ts:677
    (anonymous) @ index.ts:198
    step @ index.ts:3
    (anonymous) @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    (anonymous) @ index.ts:162
    notifySubscription @ Observable.js:135
    onNotify @ Observable.js:179
    next @ Observable.js:235
    (anonymous) @ datastoreConnectivity.ts:36
    notifySubscription @ Observable.js:135
    flushSubscription @ Observable.js:121
    (anonymous) @ Observable.js:174
    (anonymous) @ Observable.js:73
    Promise.then (async)
    enqueue @ Observable.js:71
    onNotify @ Observable.js:173
    next @ Observable.js:235
    (anonymous) @ Reachability.ts:21
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ datastoreConnectivity.ts:31
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ index.ts:162
    (anonymous) @ index.ts:161
    step @ index.ts:3
    (anonymous) @ index.ts:3
    fulfilled @ index.ts:3
    Promise.then (async)
    step @ index.ts:3
    (anonymous) @ index.ts:3
    push../node_modules/@aws-amplify/datastore/lib-esm/sync/index.js.__awaiter @ index.ts:3
    (anonymous) @ index.ts:151
    (anonymous) @ index.ts:400
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ datastore.ts:607
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    fulfilled @ jsSha256.ts:110
    Promise.then (async)
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
    DataStore.start @ datastore.ts:559
    (anonymous) @ datastore.ts:967
    step @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    (anonymous) @ jsSha256.ts:110
    push../node_modules/@aws-amplify/datastore/lib-esm/datastore/datastore.js.__awaiter @ jsSha256.ts:110
    (anonymous) @ datastore.ts:966
    (anonymous) @ datastore.ts:973
    Subscription @ Observable.js:197
    subscribe @ Observable.js:279
    (anonymous) @ MerchantContextProvider.tsx:143
    invokePassiveEffectCreate @ react-dom.development.js:23487
    callCallback @ react-dom.development.js:3945
    invokeGuardedCallbackDev @ react-dom.development.js:3994
    invokeGuardedCallback @ react-dom.development.js:4056
    flushPassiveEffectsImpl @ react-dom.development.js:23574
    unstable_runWithPriority @ scheduler.development.js:646
    runWithPriority$1 @ react-dom.development.js:11276
    flushPassiveEffects @ react-dom.development.js:23447
    performSyncWorkOnRoot @ react-dom.development.js:22269
    (anonymous) @ react-dom.development.js:11327
    unstable_runWithPriority @ scheduler.development.js:646
    runWithPriority$1 @ react-dom.development.js:11276
    flushSyncCallbackQueueImpl @ react-dom.development.js:11322
    flushSyncCallbackQueue @ react-dom.development.js:11309
    scheduleUpdateOnFiber @ react-dom.development.js:21893
    enqueueSetState @ react-dom.development.js:12467
    push../node_modules/react/cjs/react.development.js.Component.setState @ react.development.js:365
    (anonymous) @ react.js:61
    dispatch @ redux.js:222
    rehydrate @ persistStore.js:79
    _rehydrate @ persistReducer.js:61
    (anonymous) @ persistReducer.js:90
    Promise.then (async)
    (anonymous) @ persistReducer.js:89
    Promise.then (async)
    (anonymous) @ persistReducer.js:84
    p @ VM31:1
    v @ VM31:1
    (anonymous) @ VM31:1
    dispatch @ redux.js:213
    e @ VM31:1
    (anonymous) @ redux-logger.js:1
    dispatch @ VM31:1
    persist @ persistStore.js:114
    persistStore @ persistStore.js:123
    push../src/redux/configureStore.ts.__webpack_exports__.default @ configureStore.ts:15
    ./src/index.tsx @ index.tsx:21
    __webpack_require__ @ bootstrap:784
    fn @ bootstrap:150
    1 @ utils.ts:40
    __webpack_require__ @ bootstrap:784
    checkDeferredModules @ bootstrap:45
    webpackJsonpCallback @ bootstrap:32
    (anonymous) @ main.chunk.js:1
    Show 36 more frames

laclance picture laclance on 18 Nov 2020

maybe has something to do with cognito?

laclance picture laclance on 18 Nov 2020

I basically spent my entire day trying to prove there's indeed a problem(or a gap) with the underlying mechanism.

What I did was, changing AWSAppSyncRealTimeProvider.ts#L634-L653 to force it to throw a reject error (simulating Connection handshake error).
Code_2f01cYYo8Y

I encountered 2 tiny bugs after I did that, hence I've submitted a PR for it #7225.

Here's a video that shows DataStore does not recover from it:
https://streamable.com/f0zfq7

My observation:
DataStore will not perform a retry.
Calling DataStore.start() multiple times doesnt help and (this is wrong, as DS is not initialized, calling it the second time will reattempt again but it would not help if DS is already initialized) and no mutations will go through.

How does this translate to real world scenario?
When intermittent connection drop happened on a device for at least 2 second while the websocket is getting established, all consecutive jitteredExponentialRetry() will fail. When this happens, DataStore receive an error (Connection handshake error) from the subscription but it will not perform a datastore 'restart' so to speak.

You might wonder why datastoreConnectivity.status() did not inform DS the device went offline. Technically the device went offline but it was only for a 1-2 second (connection drop). NetInfo fails to catch that.

I really hope my information would be make useful to expedite the fix.

nubpro picture nubpro on 18 Nov 2020
👀1

From the way I am understanding this, DataStore is still making multiple attempts to reconnect whenever it encounters Connection handshake error (with jitteredExponentialRetry) up until the maximum delay time (MAX_DELAY_MS = 5000) is reached, in which the error is thrown. Is your expectation here that DataStore should continue to attempt the connection indefinitely when the handshake error occurs? If not, would a longer MAX_DELAY_MS provide a more comfortable buffer for the connection to be made? Let me know your thoughts.

amhinson picture amhinson on 19 Nov 2020

From the way I am understanding this, DataStore is still making multiple attempts to reconnect whenever it encounters Connection handshake error (with jitteredExponentialRetry) up until the maximum delay time (MAX_DELAY_MS = 5000) is reached, in which the error is thrown. Is your expectation here that DataStore should continue to attempt the connection indefinitely when the handshake error occurs? If not, would a longer MAX_DELAY_MS provide a more comfortable buffer for the connection to be made? Let me know your thoughts.

It is actually PubSub doing the handshake jittered retry here.

I actually think increasing the MAX_DELAY_MS would help in this case.

Would there be any implication if DataStore were to re-attempt the connection indefinitely?
Which can be done by adding Connection handshake error to disconnectionHandler(). But that would cause DataStore to restart indefinitely despite device's Wi-Fi is toggled off. Feel free to correct me.

On top of that, I think exposing an API that allows the developer to restart the connection manually isn't too bad of an idea.
This control allows the user to make the decision themselves in case DataStore is stuck.
Update: I just realize u can "restart" datastore by doing DataStore.stop() followed by DataStore.start().

nubpro picture nubpro on 20 Nov 2020

On top of that, I think exposing an API that allows the developer to restart the connection manually isn't too bad of an idea.
This control allows the user to make the decision themselves in case DataStore is stuck.

Calling DataStore.stop() and DataStore.start() would cover this, right?

amhinson picture amhinson on 20 Nov 2020

On top of that, I think exposing an API that allows the developer to restart the connection manually isn't too bad of an idea.
This control allows the user to make the decision themselves in case DataStore is stuck.

Calling DataStore.stop() and DataStore.start() would cover this, right?

Yes that is correct, it was only recently added with the introduction of selective sync.

nubpro picture nubpro on 20 Nov 2020

It seems that when DataStore does not re-attempt the subscriptions when any of the subscription has timeout (again due to network drop).

https://github.com/aws-amplify/amplify-js/blob/b7eef9dc2a03e7fa4d9a6bb7420a44b0731ecc35/packages/pubsub/src/Providers/AWSAppSyncRealTimeProvider.ts#L541-L558

nubpro picture nubpro on 23 Nov 2020

@amhinson Don't mean to be rude, but is there any updates on this?

nubpro picture nubpro on 6 Jan 2021

We are still thinking through this one, as it also relates to https://github.com/aws-amplify/amplify-js/issues/7036. To summarize, it appears that there are actually 2 separate (but related) issues within this. One being the subscription timeout regarding the default timeout of 5 minutes, and the other being how the handshake error is handled.

For the subscription timeout, this could potentially be some configuration added in the future to override the default value of 5 minutes:
https://github.com/aws-amplify/amplify-js/blob/2a53beff4ffbeeb9857dd8c144fcf950ba09f7e8/packages/pubsub/src/Providers/AWSAppSyncRealTimeProvider.ts#L143
There are certain intricacies as to why this was used, but we will treat this as a feature request for now and make sure it is accounted for with any future updates around this behavior since this is the same across all other platforms and potentially involves AppSync changes as well.

For the handshake error, this is also tough one to reason about because replicating the issue reliably has still proven to be difficult without changing the code as you mentioned. However, it seems like the main issue here is dealing with network connectivity issues when NetInfo is unable to detect and transient issues, and thus doesn't send an offline event. As I noted earlier, DataStore/PubSub does actually retry the connection multiple times until the max delay of the jittered retry is met, but apparently sometimes that length of time is not enough. You and I did previously discuss adding more time to the max delay which could probably solve most cases, but it's likely that the issue would pop up at some point in the future. I'm also not a fan of having DS retry indefinitely for the reason you provided as well.

With that said, one potential solution to the handshake error would be to emit a Hub event so that the developer can listen and retry (DataStore.stop() & DataStore.start()) based on their own app business logic.

We are still thinking through these solutions as they related to other issues, so I will keep you posted as things progress.

amhinson picture amhinson on 7 Jan 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

karlmosenbacher picture karlmosenbacher  Â·  3Comments
cgarvis picture cgarvis  Â·  3Comments
ldgarcia picture ldgarcia  Â·  3Comments
rayhaanq picture rayhaanq  Â·  3Comments
lucasmike picture lucasmike  Â·  3Comments