Lunie: Sidecar failing

[Bitcoinera]

As you may notice we needed to revert the sidecar implementation for Polkadot in the API. The reason for this is a CORS error popping up and preventing a complete load of balances, block, proposals and so on. We released too fast since this error was present in the beginning, only not locally (but on https://staging--lunieio.netlify.app/ was also to be seen, when connecting to staging-api).

Here the console logs:

Kusama not loading as a result:

I opened a PR hacking this (#5024) a bit since I noticed subscriptions might be the problem here. But I am still not sure what is causing this. Some problem between netlify and the node we're connecting to for sidecar, but we are not getting the usual error for CSP. The error itself is quite strange, since we are always querying from those API URLs, but only for Polkadot networks with sidecar it will pop up.

Still investigating and hopefully we can solve this asap, as sidecar is a great performance improvement for Polkadot

[mariopino]

Is this related to sidecar? There's no connection to sidecar from frontend.

[Bitcoinera]

Is this related to sidecar? There's no connection to sidecar from frontend.

But frontend is connected to API, and API seems to be sending different headers when reporting from sidecar. Strange but yes, it has directly to do with sidecar

[mariopino]

Yep, definitely. I remember those CORS errors when api is faling in some query for some reason.

[faboweb]

@Bitcoinera did you investigate what really went wrong?

[Bitcoinera]

Yes, I did, but locally everything went smoothly. I also mistakenly focused on CORS possibilities (headers, API configuration...) but then realized afterwards it had nothing to do with CORS. Loki was not working so I wasn't able to look at the logs in staging/production. Will check now

[Bitcoinera]

OK, so these are the logs: https://grafana.lunie.io/explore?orgId=1&left=%5B%221601330400000%22,%221601503199000%22,%22Loki%22,%7B%22expr%22:%22%7Bhost%3D%5C%22Lunie-API-production%5C%22%7D%22%7D,%7B%22mode%22:%22Logs%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D

There are tons of websocket disconnections. API was completely unusable for Polkadot networks

[faboweb]

Did you actually change the URLs to sidecar in the db?

[faboweb]

Connections via Websocket usually drop, when the queries take too long. I.e. when there are too many queries in parallel.

[Bitcoinera]

Did you actually change the URLs to sidecar in the db?

Maybe this was the issue all the time? We can try again. Although I remember checking all the urls in DB and they were all the same :shrug:

But I'm not 100% sure

[mariopino]

Sidecar seems stable rn so closing