V8-archive: Saving items with M2M or O2M takes very long

[Icepick]

Bug Report

I was facing a similar issue as directus/app#2374.

When saving an item within a collection (e.g. news collection) with many selected M2M or O2M related items the post request is very slow. The request takes between 1 to 5 min or the API even timeout with a _Cannot read property 'message' of undefined_ error message. I have the feeling, that the issue might be related to the fact that the body payload contains a huge multidimensional object with all resolved nested objects. In my case, I want to save news to display in 10 branches where each of the branches contains on average 20 advisors. The saving is getting slower and slower whenever I add more and more branches or advisors.

I have the following relations between my collections.

Collection Advisors:

Collection Branches:

Collection News:

Request

Expected Behavior

Saving would operate faster.

Actual Behavior

Saving is taking very long or even fail with a timeout/error.

Technical Details

Directus Release Version: 8.5.4

[rijkvanzanten]

@Icepick Could you send the request payload that that request made? The app should only send the changed values to the API (rather than the full data). If something is wrong there, we can optimize it in the app.

(@benhaynes I'm thinking most of the optimizations here have to come from the API, as the app (should) only post(s) the diff)

[Icepick]

@rijkvanzanten sure, I added the payload to a separate document: https://docs.google.com/document/d/1oU6ARFUaXZHut3cX_fTbC_FYk1eDGAoxHTc_-5f-kek/edit?usp=sharing

When I create a new item the provided payload was sent with the first request to the api. On creation, it sends the full data. After that, the app only sends the changed values.

Wouldn't it be more performant if the POST sends only the id references of the items that were selected in a m2m list as they are already stored in the db?

[nicam]

@rijkvanzanten we (@Icepick or me) can also supply you with a database dump to debug the problem better if that's any help. I'd prefer to send it through a PM in slack or something similar though.

[rijkvanzanten]

@arboehme I think your PR that solves https://github.com/directus/app/issues/2576 also has a huge impact on this one 🙂

@Icepick @nicam just to confirm, you're referring to selecting existing items here as well right?

[nicam]

@rijkvanzanten yes when we select to many existing items, the request gets so large, that it starts failing to insert.

It might be fixed with directus/app#2600. We'll update and give it a shot - Keep you posted

[rijkvanzanten]

I'll release that fix in v8.5.5 in a minute here

[Icepick]

@rijkvanzanten awesome, the saving performs way faster with that fix. What is still slow, is the api validation (validateManyToManyCollection) withing ItemService. When I comment it out it operates super fast. I'll open a separate ticket for the api part.

[rijkvanzanten]

@Icepick @WoLfulus is hard at work revitalizing the core foundations of the API in Laravel over at https://github.com/directus/api-next/. That should also provide a huge performance boost.

[khartigan]

@Icepick was there ever a fix for this issue? I'm still encountering this exact issue on v8.6.2.

[rijkvanzanten]

@khartigan upcoming v9 is about 5x-9x faster in saving multi-dimensional related data, check out https://github.com/directus/next to give it a spin