Openfoodnetwork: [Performance] On /enterprise/shop

[RachL]

Goal

- As a: regular user
- On page: /enterprise/shop
- I want to be able to do: access the page in less than 5 seconds.

Success indicator

In the case where a shop has 10,000 products the page can display in less than 5 seconds with regular internet connection

Part 1 - improve logging and data collection

• [x] Improve Memcache Logging - #515 (OFNInstall)

Part 2 - backend performance improvements

• [x] Many inefficient queries - OCs Cannot have Master Variants #4251

- [ ] Many inefficient queries - Spike #3853

- [ ] When a shopfront's products don't fit into cache, the shop page takes ages to load #4256

Part 3 - product pagination

• [x] Spike Impact of pagination on cache.

• [x] Move product list search server-side #4040

• [x] Move product category search server-side #4041

• [x] Pagination in enterprise/shop #4042

Barriers

• The cache solution we use to load the product list doesn't allow for a paginated view.
• We do not have an existing UI design pattern available for pagination on the platform.

[luisramos0]

I assume the solution here is to optimize the way the current "infinite scroll" is working now, correct?

[RachL]

@luisramos0 yes, we talked about pagination at the gathering. But also understanding why the infinite scroll stops at some point

[luisramos0]

ok.
i just had a quick look at a long list of products:
https://openfoodnetwork.org.uk/tamar-valley-food-hubs/shop

in this case, the products endpoint returns over 500kb in under a sec.
I'd guess these 500kb of products in json are too much for the angular code. I think we need to build pagination here and make the infinite scroll iterate through the pages.

1. Frontoffice is currently getting products from controllers/shop_controller#products, no pagination
2. Maybe we should build a new api endpoint in controllers/api/products_controller for this with pagination
3. Or re-use the backend api endpoint that already has pagination: controllers/spree/api/products_controller#bulk_products

We can build pagination in 1, create 2 with pagination, OR create a frontoffice parallel in 3.

[Matt-Yorkley]

We could start with improving the query? #3853

[luisramos0]

could be Matt, I am not sure, if you build pagination, the query may be fine for each page?

[Matt-Yorkley]

I just read your notes again. If we're returning 500kb of JSON, we're probably doing something completely crazy.

[luisramos0]

I just noticed we are not using HTTP compression are we? gzip in http can have major impact on performance... these 500kBs could be very very small if gzipped...

I dont think there's anything wrong there. it's 327 products, ~1.7kb per product.
and this product looks ok with 1400 characters ~ 1400bytes:

{  
      "id":202450,
      "name":"Asparagus - Green",
      "permalink":"asparagus-green",
      "meta_keywords":null,
      "on_demand":false,
      "group_buy":null,
      "notes":null,
      "description":"Asparagus is a truly seasonal crop, growing outdoors and being picked from April until the longest day in June. Enjoy it's fresh taste whilst you can! ",
      "description_html":"Asparagus is a truly seasonal crop, growing outdoors and being picked from April until the longest day in June. Enjoy it's fresh taste whilst you can! ",
      "properties_with_values":[  
         {  
            "id":4,
            "name":"Produced within 15 miles",
            "value":""
         }
      ],
      "variants":[  
         {  
            "id":208581,
            "is_master":false,
            "count_on_hand":13,
            "name_to_display":"Asparagus - Green",
            "unit_to_display":"250g Bunch",
            "unit_value":250.0,
            "options_text":"250g Bunch",
            "on_demand":false,
            "price":"2.6",
            "fees":{  
               "sales":"0.47"
            },
            "price_with_fees":"3.07",
            "product_name":"Asparagus - Green",
            "tag_list":[  

            ]
         }
      ],
      "master":null,
      "primary_taxon":{  
         "id":200054,
         "name":"Fruit & Veg",
         "permalink":"products/fruit-and-veg",
         "icon":"https://s3.amazonaws.com/ofn-uk-production/public/images/spree/taxons/200054/original/taxons_vegetables.svg?1482322366"
      },
      "taxons":[  
         {  
            "id":200054
         }
      ],
      "images":[  
         {  
            "id":2980,
            "alt":"",
            "small_url":"https://s3.amazonaws.com/ofn-uk-production/public/spree/products/2980/small/IMG_20190425_192048734.jpg",
            "large_url":"https://s3.amazonaws.com/ofn-uk-production/public/spree/products/2980/large/IMG_20190425_192048734.jpg"
         }
      ],
      "supplier":{  
         "id":200186
      },
      "price":"2.5"
   }

[Matt-Yorkley]

I think we do have compression enabled:

[luisramos0]

yes, I see. but not the json response for products.
maybe we can use something like this? https://thoughtbot.com/blog/content-compression-with-rack-deflater

[sauloperez]

Thanks for raising the flag @luisramos0. I also think there might be many low-hanging fruits related to performance that we might be missing. Let's not forget to check this first! I want to check out https://developers.google.com/web/tools/lighthouse/, for instance.

[Matt-Yorkley]

I just looked at that last night! :smile:

[Matt-Yorkley]

A couple of things I took from it:

• We've totally disabled browser caching to fix another issue, and that's effecting performance:

# app/controllers/application_controller.rb

  def set_cache_headers # https://jacopretorius.net/2014/01/force-page-to-reload-on-browser-back-in-rails.html
    response.headers["Cache-Control"] = "no-cache, no-store, max-age=0, must-revalidate"
    response.headers["Pragma"] = "no-cache"
    response.headers["Expires"] = "Fri, 01 Jan 1990 00:00:00 GMT"
  end

• Our CSS and JS files are really bloated. I really don't like the way all of our JS gets loaded on every page as a single file, it seems really inefficient...

• We load the google maps API library and Stripe API library from external sources on every single page, but we only need them on a couple of pages. I have a feeling they are blocking, because they are at the top of the DOM. They don't seem to get cached either.

[luisramos0]

@Matt-Yorkley

1- re disabled cache :-O

2- loaded at once and bloated are different issues:

• re "loaded" - re "I really don't like the way all of our JS gets loaded on every page as a single file"
  I think this is nice, it's 500kb of JS that is downloaded by every user only once every release (it gets a new timestamp on each release). it's cached for a year on the client side :+1:
• re "bloated" - there are multiple issues in this point, one is for example the server side js translations being included here... there's a tech debt issue for that...

3- yeah, "reduce blocking JS" is always showing up on any automated performance analysis you do in our web pages. One of the solutions is really just to move them to the bottom of the DOM, isnt it?

I run lighthouse on https://openfoodnetwork.org.uk/tamar-valley-food-hubs/shop
It rates it a 9 on a scale from 1 to 100 😭 it's a good measure for the success of this issue... we can do better than 9.

[luisramos0]

lighthouse agrees with my gzip suggestion on the json file:

with Matts point about stripe and gmaps:

but the first one is really server response time:

server response time can be improved by improving the query and/or pagination.

[luisramos0]

an extra one from lighthouse is the cache config, we are not caching the big images 😭

[sauloperez]

After the spike we'll go on with the assumption that the performance gains pagination and the queries improvements will turn caching useless. Only after these things are done we'll see whether our hypothesis was right or not.

[sauloperez]

I did a first attempt at putting together a Datadog Timeboard to follow the improvements each of these epic's issues does on the /<enterprise>/shop route. Each of them should reduce the numbers displayed there.

[lin-d-hop]

Ladies and gentlemen of the OFN community...

Now that shopfront pagination has been merged I am closing this Epic in this round of Performance improvements.

Huzzah!