Watermelondb: Initial Sync Download

[PatrickLambert1101]

I see in the docs the Sync API function pullChanges uses a fetch request to get the change in records

My question is how does this fetch request handle large amounts of data, for example if an existing users, gets a new device and has to download 500,000+ records linked to their profile? Is there a way of initialising the watermelon db from the remote DB and guarantee all the data is there?

[delkant]

Also how will this work with react dom vs react native?. In one hand with react native -- since it is a pre-built app -- if you have generic data (ie. for all users) you should be able to ship a pre-loaded db with the app (is there any documentation on this? ) but you don't have this option with a webapp, will that means we should be able to download a ready to use db from a remote location? how? or we will have to go record by record with the sync API?

[sators]

+1

[SavioScaglioni]

+1

[Skullcan]

I have a similar question/problem.

The initial sync download on my App is around 46mb of data.
The download/parse of the pullChanges is ok.
But the app crashes after the object is returned to the synchronize() method.

PS: This only happens on android emulator, but it renders the development impossible since I can't test new installations properly

Edit: tried to use _unsafeBatchPerCollection in the synchronize() method, but with no success.

[radex]

@Skullcan 46MB is quite a lot. There's many tricks you can use if you can't get this size down. _unsafeBatchPerCollection is one such trick. Another is to enable android:largeHeap="true" in your manifest

[Skullcan]

Hi @radex thanks for the quick response.

Yah, we are going to change our approach, probably.

I tried the tricks you mentioned with no success. I keep getting partially executed syncs.. next time we try to sync.. I get diagnostics errors and the app crashs again.

I was reading about Batch seeding in #410 and I think we should consider this to fix our problem. Especially because we can't reduce the size of the app data atm.

Any other insights would be appreciated.

Here's the scenario:
Initial load is 46mb.
The sync works on top end phones, iPhones and S10's i.e.
I thought it crashed only in the emulator, but it's crashing on low end phones also.
Tried android:largeHeap="true" and _unsafeBatchPerCollection, no success.

Thanks again.

[radex]

give me more information: what's the cause of crash, what's the memory stats before sync, during, at crash. maybe that will be enough for me to give you guidance.

[Skullcan]

give me more information: what's the cause of crash, what's the memory stats before sync, during, at crash. maybe that will be enough for me to give you guidance.

Hi @radex

Here's the infos you asked.

Cause of the crash as per android studio's logcat:
OOM: [] reason = Max heap size was exceeded (1 from category: vm_allocate_category), numCollections = 920, heapSize = 538185728, allocated = 534975713, va = 348127232.

Memory usage in idle: avg 235mb
Memory usage during sync: min 330mb, max 520mb
Memory usage at crash: 450mb
All the values are approximated and it's source is the android studio profiler.

Here's the end of the logcat from the moment watermelon started processing the sync info.
logcat_crash.txt

Don't know if all this will be helpfull.. I'm working at some kind of workaround this rn.

Thanks again for your time.

Edit: Forgot to mention the emulator I used for testing purpose is Android Studio Nexus 5 with it's default configuration.

[radex]

try tweaking this number down and see if that's enough to avoid the crash:

https://github.com/Nozbe/WatermelonDB/blob/master/src/sync/impl/applyRemote.js#L242

[Skullcan]

Hi, thanks for your suggestion.
Sadly it not worked :(

I tried to lower the number per batch progressively till 500...
The result is always the same. The debug build, after roughly 20-22k records are processed, crashes with the same error mentioned above..
Tried other Emulators too. But it had no success either on Pixel 3, Nexus, and the same using Genymotion emulators.

I then tested an internal production build in real devices.
The change helped a bit. Albeit very slow, it works now on phones that was crashing previously.

[radex]

I have some ideas how to dramatically improve this situation, but since we're not running into this limitation at the moment, I probably won't implement it in the very near future. But in the meantime, consider hacking the initial sync a little bit by passing data in chunks -- i.e. first sync only saves part of the data, then run it a few more times, until you have all the data.

If you have the skills to get your hands dirty with C++ and JNI on Android, then you could also help me finish Android JSI integration, which would be the biggest step towards permanently fixing this situation

[Skullcan]

Alright, I understand and appreciate your help in this case.

Unfortunately I don't have the skills nor the time available to help with this.

I'll do what you suggested tho, I think that's the best approach atm.

Thanks very much for your time.

We've been using 🍉 for about 1 year now, and it's very nice and easy to work with. Keep it up!