Aws-mobile-appsync-sdk-js: Best practice for real-time data syncing

[jmparsons]

In the case of many inserts happening at the same time I've noticed the graphql subscriptions not catching many of them. A simple example would be to take the provided sample schema and project, then spam the add comment button on an event - I've canned some Math random message to auto generate a comment on click. I've tried to increased the read and write capacity units, but the results are the same. Spamming up to three comment inserts will work just fine, but anything more and they don't get added - two browsers side by side showing the same event page.

[ryan-mars]

@jmparsons This is very interesting. Do you have a demo you could share?

[jmparsons]

I really did just create new sample schema and downloaded the starter app then ran it with the credentials and changed the create comment to use a canned message instead of the text field.

[ryan-mars]

@jmparsons I just created the sample schema and downloaded the sample app.

In NewComment.js after:

    await createComment({
      ...comment,
      eventId,
      createdAt: moment.utc().format()
    })

I added this.

Promise.all(
  [...Array(10).keys()].map(e =>
    createComment({
      content: `${e}`,
      eventId,
      createdAt: moment.utc().format()
    })
  )
)

I get my comment, plus 10 additional numbered comments. Although they do arrive out of order. I'm using Firefox and Chrome and watching it happen to both. It's not instantaneous but it resolves in under 3 seconds.

[jmparsons]

@ryanwmarsh Try to stagger the calls. I changed:

const { comment } = this.state;

to:

const comment = { content: `hi-${Math.random().toString()}` };

Then placed the add comment button above the list so I could spam click it:

{[].concat(items).sort((a, b) => a.createdAt.localeCompare(b.createdAt)).map(this.renderComment)}
<NewComment eventId={eventId} />

<NewComment eventId={eventId} />
{[].concat(items).sort((a, b) => a.createdAt.localeCompare(b.createdAt)).map(this.renderComment)}

[ryan-mars]

@jmparsons I did notice a weird bug where my comments are being duplicated once GraphQL tries to sync with the API. Wait for it.

[ryan-mars]

@jmparsons Now I'm getting some really weird behavior. Try this and submit one comment after the other. Then refresh the page. The second set of comment+numbers are disappearing on me.

    await createComment({
      ...comment,
      eventId,
      createdAt: moment.utc().format()
    })

    await Promise.all(
      [...Array(10).keys()].map(n =>
        createComment({
          content: `${n} ${Math.random().toString()}`,
          eventId,
          createdAt: moment.utc().format()
        })
      )
    )

[jmparsons]

@ryanwmarsh Yeah same thing for me.

[ryan-mars]

@jmparsons The deeper I dig and more permutations I try (run mutations from AppSync console, etc...) the more weirdness I'm finding. Either I'm using AppSync wrong or there are problems with the AppSync SDK. I'd like to find out which because right now I can't put something like this into production.

[jmparsons]

@ryanwmarsh I'm in the same boat. If the syncing isn't spot on I'm going to have to resort to polling. I can sort and filter the results, but the actual data has missing inserts.

[undefobj]

@jmparsons @ryanwmarsh we're going to replicate this. It's worth noting that AppSync does NOT guarantee the order of messages, so that piece you'll need to do yourselves.

Jon could you ping me via my AWS email address? I'd like to look at your request IDs too. Ryan if you want my email you can DM me on Twitter.

[jmparsons]

@undefobj on it.

[dennari]

Any news on this? I'm considering AppSync mainly for real-time data syncing and this looks worrisome.

[undefobj]

@dennari we are still tracking this and have a design, but it's a very particular edge case for very high velocity writes on sync that we need to implement carefully to ensure data integrity. Are you running into this in production? It shouldn't be common for most use cases, as members such as @jmparsons had to write code with many elements in a loop to reproduce.

[arelaxend]

Hi, I have found the same issue, I have a subscription and many mutates to update state of the app (to do a loading feature), and some mutations do not trigger events. Any update? i am testing with amplify 1.0.8 and amplify-react 2.0.0

[elorzafe]

@jmparsons @dennari @arelaxend @ryanwmarsh have you tried latest aws-appsync(1.7.0), high velocity writes shouldn't be an issue right now. Let me know how it goes. Otherwise I will close this issue in a week from now.

Thanks!

[ryan-mars]

Thanks, you can close this.

[jmparsons]

I'll give it a try in a bit, but thank you in advance!