Parse-sdk-ios-osx: LocalDatastore + Query caching

[SaifAlDilaimi]

Hello guys,

I'm working on a App that is similair to AnyPic and have some question. I think its known that as you enable LocalDatastore the application crashes on every cache policy.

Well since the SDK is now open sourced I was thinking if you guys could explain me why I can't use LocalDatastore in combination with cache queries.

In some places I dont want to use the datastore but instead the cache because the data is not that relevant. Now I can't, right?

Maybe I'm to naiv but wouldn't it be perfect to add a Policy like "NetworkThenLocalDatastore" / "NetworkElseLocalDatastore" or "LocalDatastoreOnly" ?

I would like to get some hints how I can use both in combination and why the Policies didn't were implemented? In my opinion it would perfect.

[nlutsenko]

The biggest reason behind disabling query caching with Local Datastore is actually behind the fact that Local Datastore is persistent and guaranteed to have an object if you pinned it to it.
Whilst the cache is cache and could be evicted at any given point of time.

The flow that I would suggest using is something like this:

• Fetch from local datastore to display results
• If no results found - fetch from network
• Fetch from local datastore to display results

The overall structure is quite simple - you always fetch from local datastore to show the results that you want to make sure are available offline. If no results are available - you refresh from the network (which persists any object that was saved to local datastore), then you re-fetch from local datastore.

The other important piece is that local datastore is not guaranteed to have all the results for your query, but rather will have only objects that you explicitly pinned to it.

Let me know if this helps at all as well as I am all in for finding a caching solution using local datastore that will behave closer to query caching.
We had few ideas around auto-expiring pins (close to LRU cache, but purely in Local Datastore).

[SaifAlDilaimi]

first, thank you for your detailed answer.

But now I ask myself why we cant make it so that we add new Policies to the ones existing but for the Datastore? This would solve the issues you just described, right? Maybe Adding a method where a user can also say that he wants this explicit query result to be pinned.

btw, i know how much work you guys have with these stuff in github and I just wanted to thank you for the support!

[SaifAlDilaimi]

@nlutsenko is there anything new that you can tell me?

I feel like its a definition problem. I would like to have some queries to having the caching workflow but with the features of the LDS.

[SaifAlDilaimi]

Hey @nlutsenko,

I had some thoughts about the Problem.. I think I have a idea, can you give me your thoughts?

-> Changing/Adding the method [query setCachePolicy] to [query setStorePolicy]

New Policies (adding to the existing ones)

• LocalDatastoreThenNetwork
• NetworkElseLocalDatastore
• LocalDatastoreElseNetwork
• LocalDatastoreOnly

Declaring the use of LocalDatastore
When quering you can set this specific query to be pinned to the LDS for example: [query toLocalDatastore]. This method would use the [PFObject pinInBackground] method. I'm just not sure if there are problems to migrate the method to the PFQuery Class.

What do you think?

PS: I'm sorry if that's annoying but I feel like its a really important aspect.

[nlutsenko]

Fantastic proposal, I agree with introduction of new policies and usage of query.storePolicy.
The only one that is probably missing - NetworkOnly policy, which would be set by default.
This would simplify a lot of code, as loading from LDS, then network is such a common operation.

Here are my thoughts on every aspect:

Store Policy

• LocalDatastoreThenNetwork - this would work the same way as with query caching, meaning it will only be useful and work with 2 invocations of the same block/callback. While still useful, the intended way of using LDS is to tie your UI completely to it, and seed it from network from time-to-time.
• LocalDatastoreElseNetwork - this is interesting one, as queries against local datastore always succeed. This might treat no objects returned as failure, making it quite useful.
• NetworkElseLocalDatastore - useful and good, would work the same as query caching
• LocalDatastoreOnly and NetworkOnly - exist via slightly different API.

Declaring use of Local Datastore

toLocalDatastore is useful, but taking into account that it would use the same pinInBackground and won't impact performance, as well as the fact that building this piece yourself by subclassing a query is soo easy - I don't think this would be very useful for a first-party API.

cc @richardjrossiii @grantland @hallucinogen
Guys thoughts about this API applicability to other platforms?

[FreudGit]

Love the idea.
LocalDatastoreThenNetwork = If this can also give a way to get local data, then get (only) new stuff from Network. Wow.

[SaifAlDilaimi]

Hey guys,

@nlutsenko I'm glad you like it. I would like to give my thoughts:

LocalDatastoreThenNetwork

this would work the same way as with query caching, meaning it will only be useful and work with 2 > invocations of the same block/callback. While still useful, the intended way of using LDS is to tie your UI completely to it, and seed it from network from time-to-time.

It does behave like query caching but the second callback should just be invoked when new Objects are available.

LocalDatastoreElseNetwork
Agree.

NetworkElseLocalDatastore
Agree.

toLocalDatastore is useful, but taking into account that it would use the same pinInBackground and won't impact performance, as well as the fact that building this piece yourself by subclassing a query is soo easy - I don't think this would be very useful for a first-party API.

That's true but still would be only possible with modifying and adding logic to a subclass of PFQuery.

In my opinion this approach would solve the barrier between Caching and LDS.

While still useful, the intended way of using LDS is to tie your UI completely to it, and seed it from network from time-to-time.

I wouldn't see that as the only intended way. It's a LocalDatastore and I would use it to save not only things that are UI-Linked.

[SaifAlDilaimi]

@FreudGit

Love the idea.
LocalDatastoreThenNetwork = If this can also give a way to get local data, then get (only) new stuff from Network. Wow.

It is, I always struggle to build this when you are building a Newsfeed. This Store policy would solve so many things..

[devmsh]

any updates about this issue?

[jadsonlourenco]

+1

[nitrag]

+1, this should be a necessity.

[SaifAlDilaimi]

Any idea what it would take to implement this? Would love the take this further.. I could need some hints and a opinion if this is something that you guys want.

[nitrag]

Opened a bounty on this, please contribute so we can get this much needed feature.

https://www.bountysource.com/issues/26840483-localdatastore-query-caching

[thisfiore]

Really happy to see it on the roadmap.

Next week I'm going to support offline mode for data and files in my app.

Do you think I can achieve this with local datastore + native files saved with references in datastore?

[flovilmart]

@thisfiore is this a feature you would be willing to work on?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you believe it should stay open, please let us know! As always, we encourage contributions, check out the Contributing Guide