Netlify-cms: Slow search

[eele94]

Is your feature request related to a problem? Please describe.

Feature Request.
When having a large collections, it is hard to find the right item.

• Search is slow.
• No abilty to sort items based on date or something else.

Describe the solution you'd like

• A faster search, maybe option to load all content at once or cache after first search.
• able to set sortColumns in the admin.config

Describe alternatives you've considered

Haven't thought about anything else.

Additional context

NetlifyCMS is awesome!

[talves]

_Note to self_: It would really be good for the community to setup an example of a large set of data in a collection to test against.

Paging will definitely be the first step. Filters (available now) could help somewhat maybe. Maybe even splitting up the collection into smaller collections for edit.

Sorting issue is tracked here

To get some feedback on how other people are handling large collections, you might want to bounce the issue off the community for ideas of how they are handling it.

[Freysi21]

Is anyone having issues with the search not working most of the time?

I keep getting a 429 status code when I run a search. I'm currently hosting against bitbucket.
I've been able to get result if put in search string send it and then start erasing it (backspace). Pretty weird...

[erquhart]

Searching is local, but for paginated backends we grab all pages of entries first - sounds like we're making too many requests too quickly for the Bitbucket API.

[Freysi21]

Been able to get results more consistantly by manually entering the url+"admin/#/search/"+searchstring in my browser if that helps.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mushkin-v]

The same problem - too many requests too quickly for the Bitbucket API... Any workaround for this?

[eele94]

I made a page where you load all sites in a Javascript variable and make a custom search bar that filters through that variable

[mushkin-v]

I made a page where you load all sites in a Javascript variable and make a custom search bar that filters through that variable

@eele94 Thank you, I know how to make a page, to load all info and to make a search on it, but how to put this page under CMS(bitbucket auth etc)?

[eele94]

You can make the page with JSON output and do a request from netlifycms to the generated page. You have to fork netlifycms or add to /admin/index.html a small script to load custom search logic

[mushkin-v]

You can make the page with JSON output and do a request from netlifycms to the generated page. You have to fork netlifycms or add to /admin/index.html a small script to load custom search logic

Thanks @eele94, for such a quick answer. I really don't want to fork CMS, but an option with /admin/index.html sounds interesting. So I can just put index.html in /admin folder, and CMS will override it? Maybe you know a link where a can read more about this? I don't remember such abilities in CMS docs((

[eele94]

No add custom script to the /admin/index.html that does a JSON request to your custom searchdata page. You can hook the custom code for search bar into the dom.

[tayelno]

Had same issue I have site with 5k+ pages (added manually without using netlify-cms) in the same collection saved as frontmatter, tried adding editorial workflow, it sent many requests to Github, which actually blocked my github account for the whole day :(

[tayelno]

after reading the architecture page on contribution guide here https://www.netlifycms.org/docs/architecture/

this section confused me

About metadata
Netlify CMS embraces the idea of Git-as-backend for storing metadata. 
The first time it runs with the editorial_workflow setup, it creates a new ref called meta/_netlify_cms, pointing to an empty, orphan tree.

Actual data are stored in individual json files committed to this tree.

Does this mean each page is being returned by a single request, can't we just list all the files using github gitlab apis such as using GET /repos/:owner/:repo/git/trees/:sha once then search on the client-side ?

[erezrokah]

@tayelno The tree API will list the files, but not their content... We might be able to use the GitHub GraphQL API to list files and retrieve their content in one request.
As for hitting the limit of GitHub API, adding pagination to GitHub might help with that (avoids retrieving all the files on load), though not when running a search.
The challenge is similar to https://github.com/netlify/netlify-cms/issues/54, where we might be able to use the diffing API to just get the changed files.

[erezrokah]

There is an improvement to the search that might be easier to implement.
At the moment the CMS will only display results for the search once all entries are retrieved.
We could display intermediate results based on the pages loaded so far and once all entries are retrieved update the view again.

[erezrokah]

Closed in https://github.com/netlify/netlify-cms/pull/3494