Openfoodnetwork: Reduce performance impact of /shops

[sauloperez]

What we should change and why (this is tech debt)

We need to reduce the impact the /shops page is having on our servers load by reducing its response time with quick and easy optimizations. It's holding its position as the worst app endpoint in the last month.

Context

With the increased demand and the scarcity of resources in FR and UK servers this page's response times are going nuts, and this causing the DB to be on :fire: as well.

Today, the percentile 95 reaches 15secs sometimes, like this request. And it's getting worse.

Count of requests taking more than 10s in the last 4h for UK and FR together.

Even the latency distribution is taking a very weird and far from good shape.

Impact and timeline

Any improvement in this page will have a big impact on the server load. Doing more with the same resources it's a good way to scale-up the system, isn't it?

[sauloperez]

I tried an aggressive caching strategy using Rails page caching but that won't work due to the dynamic nature of this page. Even if we go with other lower-level strategies dealing ourselves with cache invalidation, I'm afraid the complexity will be pretty big and it'll take time.

My personal conclusion is that we'll need to work pretty soon on refactoring /shops if we want to survive this :chart_with_upwards_trend: As per @matt-yorkley’s insights, there are no quick wins there unless we :fire: the tech-debt.

[Matt-Yorkley]

the percentile 95 reaches 15secs sometimes

It's worse than that @sauloperez ... :grimacing:

:fire::fire::fire:

[sauloperez]

OH MY F* GOD :see_no_evil:

[Matt-Yorkley]

We can have another look at the data on this after the new release is deployed.

[sauloperez]

This is sweet! :heart:

Stats in the last two days

[daniellemoorhead]

Yay! Can we consider @sauloperez's stats update the equivalent of a #prod-test and this card can be closed now? Or is there a specific test to do?

[sauloperez]

Yep, we can close

It's holding its position as the worst app endpoint in the last month.

It has fallen down to the 4th position :v: great job @Matt-Yorkley :clap: