Netlify-cms: Relation widget performance optimization

[jozsi]

Hello,

I have a list of categories nested with lists of products, all of them beign relationship widgets. When I open the CMS editor, it fires thousands of XHR requests for about 150 relations. Is it a possible bug?

First I thought that having the lists collapsed, it will only start the API calls on expand, but later I realized it's not just the autocomplete selector that needs the data, but it's also used for fieldsMetaData of custom preview templates and possibly other internal stuff.

I suppose @ncwidgets/file-relation of @d4rekanguok could come handy for the categories part (~20 relations), but I don't think I could do the same for the products because there can be many relations of the same product in different positions/order.

What's the best approach to handle this?

Thank you!

Example code:
config.yml

collections:
  - name: category
    label: 'Categorie'
    folder: 'site/content/category'
    create: true
    fields:
      - { label: 'Nume', name: 'title', widget: 'string' }
      - label: ID
        name: id
        widget: ncw-id
        prefix: c
  - name: food
    label: 'Food Item'
    folder: 'site/content/food'
    create: true
    fields:
      - { label: 'Produs', name: 'title', widget: 'string' }
      - { label: 'Descriere', name: 'description', widget: 'text' }
      - { label: 'Pret', name: 'price', widget: 'number', valueType: 'float' }
      - label: ID
        name: id
        widget: ncw-id
        prefix: p
  - name: 'pages'
    label: 'Pagini'
    files:
      - file: 'site/content/pages/menu.md'
        label: 'Meniu'
        name: 'menu'
        fields:
          - {
              label: 'Categorii',
              name: cats,
              widget: list,
              collapsed: false,
              fields:
                [
                  {
                    label: 'Category',
                    name: 'cat',
                    widget: 'relation',
                    collection: 'category',
                    valueField: 'id',
                    searchFields: ['title'],
                    displayFields: 'title',
                  },
                  {
                    label: 'Produse',
                    name: prods,
                    widget: list,
                    collapsed: true,
                    fields:
                      [
                        {
                          label: 'Produs',
                          name: 'prod',
                          widget: 'relation',
                          collection: 'food',
                          valueField: 'id',
                          searchFields: ['title'],
                          displayFields: 'title',
                        },
                      ],
                  },
                ],
            }

menu.md

title: Meniu
url: /menu
cats:
  - cat: c-Q4S3rXcP8
    prods:
      - prod: p-ZzpgfAY3H9
      - prod: p-ctjz-Petp_
      - prod: p-yT_0exoiIPM
      - prod: p-f-Bse1yGlLd
      - prod: p-FqK42Z01aDo
      - prod: p-Lg2gAVOYCLG
      - prod: p-wm_LskJ4l2y
      - prod: p-NLuQ5CpqrKO
      - prod: p-rb2U5xzRQ9u
      - prod: p-aYzD4UyPEGS
      - prod: p-77gTOlo9cE_
      - prod: p-EJzOzTcYxiy
      - prod: p-TkNHkNK3yck
  - cat: c-CxIEUOV_qg
    prods:
      - prod: p-jH2snKpKetP
      - prod: p-IAjA7nKuzV2
      - prod: p-dVpX-Fvke_S
      - prod: p-qoFgPloLL4C
      - prod: p-PLjLIgigz2f
      - prod: p-ebGVWf3b7m5
      - prod: p-Puun2DbjMqQ
  - cat: c-LYXQCfxCCc
    prods:
      - prod: p-O2pIurx58s_
      - prod: p-kFq02zA8lBK
      - prod: p-WNXOkUgW_iD
      - prod: p-BaPIrMvgR-L
      - prod: p-FepR5UXbHS_
      - prod: p-EpftQX90Q9x
      - prod: p-KEI72VYVw2f
      - prod: p-Zw70OfyeBG4
      - prod: p-TPKVJgMIZCu
      - prod: p-nNE2RozZdwu
      - prod: p-Puun2DbjMqQ
  - cat: c-oG7hWOsgPb
    prods:
      - prod: p-1mR5bKQoQRe
      - prod: p-kT6LbmEa4gW
      - prod: p-gF49J5n07Bf
      - prod: p-BiQ9aykEANm
      - prod: p-Lpj7UceiyIX
      - prod: p-nkBW47HVF8l
      - prod: p-66_cb-Xfccy
      - prod: p-zKSkoHnJYmx
      - prod: p-WS0XufEkl80
      - prod: p-oXMqrNLL0UY
      - prod: p-nKypslvOItG
      - prod: p-FQZSwRlpm0K
      - prod: p-CD_bTCUiDF2
      - prod: p-TdBgKu8t-wf
      - prod: p-2Oll474uHW2
      - prod: p-o67XgsFvpDy
      - prod: p-Osml6aA32S8
      - prod: p-Puun2DbjMqQ
  - cat: c-dT_ElDQeOv
    prods:
      - prod: p-TgmQ-1zJztl
      - prod: p-Puun2DbjMqQ
  - cat: c-BKz5xPZ9Vs
    prods:
      - prod: p-svsOwrkL5Dv
      - prod: p-qgIfY2GYaWS
      - prod: p-tGbG62wun3R
      - prod: p-6Rdgv4tKIyY
      - prod: p-qnPd-OEQnrC
      - prod: p-Puun2DbjMqQ
  - cat: c-6ydatGrEiB
    prods:
      - prod: p-zcMThAmt4pt
      - prod: p-OpYTqZ69KyF
      - prod: p-rU3s-VDKVpl
      - prod: p-iEzbpKZzJgc
      - prod: p-8iQEiLB8VqN
      - prod: p-_VIEjkG2tuC
      - prod: p-Kr9pXwybS73
      - prod: p-ecW_-hMifKI
      - prod: p--HFlB3zs3k2
      - prod: p-JqqE16Ft-Ak
      - prod: p-foFKWevcRrL
      - prod: p-5a2XlPRfetP
      - prod: p-qLbgK7-viWS
      - prod: p-1ovzRFaadQO
      - prod: p-PzG-ySobP1O
      - prod: p-mNgn1OggkJn
      - prod: p-uZ1qxanV_2G
      - prod: p-Puun2DbjMqQ
  - cat: c-W0xIadrzqa
    prods:
      - prod: p-p6vQnGT9sNZ
      - prod: p-A9E3mhBbnAN
      - prod: p-xG4PoKjlvSd
      - prod: p-VKXZQzXA4wt
      - prod: p-X-9NkHfPQ4F
      - prod: p-QwQkLpFoDms
      - prod: p-Puun2DbjMqQ
  - cat: c-bAClA7B3kU
    prods:
      - prod: p-wIDQvmY7qck
      - prod: p-bAvfKGXUs8L
      - prod: p-jMrfYv8_w0z
      - prod: p-fXTwjEQXkO1
      - prod: p-Q1A3dcyE42x
      - prod: p-P0WuD3gh5gd
      - prod: p-MIJcHX8RwkF
      - prod: p-T2zzQbJ4hHI
  - cat: c-l3bAVtyfum
    prods:
      - prod: p-n3LsJOXRjWv
      - prod: p-rqWQ4rDIV-J
      - prod: p-UBqbyZ1QRr8
      - prod: p-cFGJYfgNm5l
      - prod: p-LcLnjasE95f
      - prod: p-45xoZlVIkZU
      - prod: p-ryzJLdzQtOi
      - prod: p-MevmP-4oUip
      - prod: p-nEb3s1jY1UQ
      - prod: p-C37NJFjRyYK
      - prod: p-Puun2DbjMqQ
  - cat: c-hh_cVKFlS1
    prods:
      - prod: p-bAvfKGXUs8L
      - prod: p-MIJcHX8RwkF
      - prod: p-Q1A3dcyE42x
      - prod: p-P0WuD3gh5gd
      - prod: p-QiuaKRmvi-h
      - prod: p-wIDQvmY7qck
      - prod: p-fXTwjEQXkO1
      - prod: p-jMrfYv8_w0z
      - prod: p-y8ExQKMTyzv
      - prod: p-GifzJjSDrrs
      - prod: p-T2zzQbJ4hHI
  - cat: c-mBYSlL9KZC
    prods:
      - prod: p-YTBVoKK-ADw
      - prod: p-s7V7RtwRuQ6
      - prod: p-Ye3l9T4fzY8
      - prod: p-ONTAZs8lgK2
      - prod: p-dl0NhduXMLU
      - prod: p-i1xOYp_-tna
      - prod: p-Puun2DbjMqQ
  - cat: c-P_iD-usBeu
    prods:
      - prod: p-NwKwMVBhzD5
      - prod: p-lm7WT_M7Upy
      - prod: p-yrEigmdoegB
      - prod: p-Onqu0_VBwbu
      - prod: p-khtU0vUOH6f
      - prod: p-s4eIxAuRR0c
      - prod: p-OIQmQxbSwJw
      - prod: p-oTUv9g-yaeJ
      - prod: p-2hKhNGrHHML
  - cat: c-rRdRkxscyy
    prods:
      - prod: p-f-Bse1yGlLd
      - prod: p-WyYoIsgilb3
      - prod: p-D3TYhAwyAx_
      - prod: p-TnPXuhJgO1e
      - prod: p-bt6kkO0JUkO
      - prod: p-Jm7gYwmlPom
      - prod: p-4K3miXQMivC
      - prod: p-3NIJbl9H1zh
      - prod: p-dbNuShfPwp9
      - prod: p-A3nLrqpRD0j
      - prod: p-P0WuD3gh5gd
      - prod: p-b7QQk1HhnK3
      - prod: p-Puun2DbjMqQ

---

[barthc]

@jozsi what's your CMS version?

[jozsi]

@barthc, I use the latest versions:
netlify-cms-app 2.12.12
netlify-cms-core 2.26.0
netlify-cms 2.10.48

[barthc]

@jozsi I assume you are on github backend. If so then the subsequent requests you are seeing are actually loading from the cache after the initial listAllEntries call and should not affect performance.

[jozsi]

@barthc: I use the GitHub backend, yes. But it makes 17.5k requests, when they are cached, still takes 2 minutes to load and meanwhile it slows down the whole browser (Chrome on latest gen i7 CPU and NVMe disk).

Maybe a little memoization could help?

[barthc]

Between how many published entries do you have in the food and category collection? It should load all entries on first initial call.

[jozsi]

I've 15 category and 123 food entries. I'll add you as a collaborator to the repository & netlify app, if it helps to reproduce. Feel free to fork it.

PS: I am currently using the the @ncwidgets/netlify-cms drop-in that's basically a wrapper netlify-cms plus two widgets (@ncwidgets/id and @ncwidgets/file-relation), but I tried with the plain netlify-cms package as well.

[barthc]

@erezrokah I think this issue is related to #3584, do we fall back to the previous approach for github backend?

[erezrokah]

I'll have to investigate, but the previous approach will be to get those entries from IndexDB (you won't see any requests but it would still access the browser storage).
Not sure it will be faster, but again I'll have to test it.

[jozsi]

Also, if the cache is not yet built or disabled, some requests even get cut off by Chrome with ERR_INSUFFICIENT_RESOURCES, others by GitHub with rate limiting errors. They ended exactly at 30 minutes with almost 60k requests, but the last couple thousands were all rate limiting ones:

[erezrokah]

Thanks @jozsi, I think the solution for it would be to allow relation widget for file collections and have a single file with a list widget, instead of multiple files per category/product.
We have a PR for that in progress https://github.com/netlify/netlify-cms/pull/3717

[jozsi]

Yep, that's what I am plan to do for the categories. Been waiting for the aforementioned PR to be merged, but I'll try with the custom widget and see how I could implement it for products as well, in a Hugo friendly way (I used a Hugo template from Netlify for this project, to see how it compares to Gatsby that I like a lot - and will stick to it in the future).

[jozsi]

I can confirm that using file collections has made things snappier. Can't wait for #3717 to be released (along with #3616)