Lunie: Notifications query making DB crash

[Bitcoinera]

This is not so much a bug but a description of a very strange realization.

A simple notifications query to get the notifications for one account, like this one:

query MyQuery {
  notifications(limit: 10, where: {networkId: {_eq: "cosmos-hub-mainnet"}, resourceId: {_eq: "cosmosvaloper10nzaaeh2kq28t3nqsh5m8kmyv90vx7ym5mpakx"}}) {
    eventType
    resourceType
    created_at
    data
  }
}

When applied for all validators (parallelized with Promise.all, of course) crashes DB. Trying it in the GraphQL Lab in Hasura it is also possible to notice that it always takes a couple of seconds to load, so it seems to be a CPU demanding query.

Solutions I propose:

1. store notifications together with validator in DB (validatorprofiles), every time a notification with a resourceId which is a validator occurs. So then we don't need to do this query in the notifications table but get it directly from the validator profile.

2. have scriptRunner querying in batches for validator feeds (i.e., latest validator notifications) in the background, as a separate instance, and store them in a new table (validatorFeeds), which we would query from API.

We could also mix some ideas from these two solutions, like having a separate table validatorFeeds, but store from API when the notification occurs, for example.

Looking forward to you ideas!

[jbibla]

thanks ana!

just to add some ideas to the mix:

• we could store notifications differently in a way that's optimized for the types of queries we want to make
• maybe we should bump the resources on the instance?

---

can you help me understand why we need to be able to parallelize these queries?

[Bitcoinera]

can you help me understand why we need to be able to parallelize these queries?

The query is already parallelized with Promise.all :shrug: That didn't help, this is what I wanted to mean

[jbibla]

we'll need @faboweb to weigh in here

[faboweb]

I could imagine the problem comes from parallelized with Promise.all. Instead it should be 1 query returning the data for all validators. this can be done via the _in operator (https://hasura.io/docs/1.0/graphql/core/queries/query-filters.html#list-based-search-operators-in-nin) or you send multiple gql queries in 1 request. Hasura should handle the rest. But not do several requests in parallel.

[Bitcoinera]

Cool, it works great! I am going to do the same for the validator profile query (also DB dependant)

[faboweb]

not doing it right now but #5081 will reduce the need for the validator profiles