In WordPress 5.0 Beta 4 in the Gutenberg editor, there is a small delay when expanding the Categories and Tags metaboxes. The Categories metabox has 20 categories and every time the box is expanded, there is a second or two delay before they're displayed. It appears as though the list is regenerated each time the box is expanded. This also happens with tags. While the delay is not lengthy, it's long enough that it's noticeable and annoying. In the Classic editor, the categories are displayed almost instantaneously.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Instantaneous display of categories and tags similar to how it works in the Classic editor.
Indeed I'm seeing the same thing: https://tinytake.s3.amazonaws.com/pulse/jcasabona/attachments/9246647/TinyTake15-11-2018-12-25-41.mp4
OS: Windows 10
Browser: Chrome
Version: 70.0.3538.102 (Official Build) (64-bit)
Every time you open the categories box, Gutenberg is querying the Rest API

To me, this is the reason, why you experience this problem. Same problem occurs also with the tag search.
Related to #6723 Provide loading UX when UI is dependent on an API request
Probably I am just not too deep into the related discussions, but why do you need to load the categories again, once you have them loaded?
The only advantage I would see is, if in the meantime new categories would have been created, but as your categorieslist (101, 201, 301,...) grow this comes with quite some downsides.
Edit:
I would have expected the loading of the terms to be done via the state management, but it seems, it is not (yet):
https://github.com/WordPress/gutenberg/blob/3caec34c7bf7730427855c2b50f780ff07e87924/packages/editor/src/components/post-taxonomies/hierarchical-term-selector.js#L190-L219
https://github.com/WordPress/gutenberg/blob/3caec34c7bf7730427855c2b50f780ff07e87924/packages/editor/src/components/post-taxonomies/flat-term-selector.js#L76-L92
Edit:
I would have expected the loading of the terms to be done via the state management, but it seems, it is not (yet):
The reasons are documented in #11643
In order to move away from reloading Categories and Tags on each request, we'll need to solve that linked issue first. Closing in favor of #11643
Most helpful comment
Every time you open the categories box, Gutenberg is querying the Rest API

To me, this is the reason, why you experience this problem. Same problem occurs also with the tag search.