Realm-js: Please provide asynchronous query/update API.

[acrazing]

Currently, the API to query(realm.objects('table')) or update(realm.create) documents is synchronous, this will lead to the javascript thread to be blocked.

It is inefficient if the table is big enough.

[kneth]

@acrazing Is your use case primarily React Native? When is a table large?

[acrazing]

Yes we are developing a chat application.

[bmunkholm]

@acrazing Is it from a theoretical viewpoint or do you have concrete practical issues with queries taking too long?

[allenhsu]

We're using realm-js for a new react native project, we haven't met any practical issue yet, but we're also looking for an async version of query APIs.

I noticed in rpc.js it's opening a sync XMLHttpRequest. Although it's a local RPC call, but it's still a network request rather than code execution. IMPO, it'll be nice to make async request and provide Promise based query APIs. It's also more intuitive to me since fetching data from server are all async and returning Promises.

[wilomgfx]

as @allenhsu mentioned and this makes chrome debugging unusable. If i try to debug my app using realm with vscode debugger (uses chrome debugger) it takes about 4 mins for my app to load. Thats ridiculous.

[bmunkholm]

@wilomgfx We get the debugging issue and consequences. But let's keep the two issues separate. We already got an issue for the debugging experience.

[markhomoki]

I think the async API would be useful for this library. In my use case, I am developing an offline-first mobile app, whereby the user has to do a manual sync with an online database to receive the most up-to-date data. My users usually use the app where there is no mobile internet or wifi connection. The online database has 36 tables and all together ~300k rows. Currently, I am looping through the tables first and then through the rows and inserting one-by-one. At the moment the whole sync process takes around 50 seconds, which is not that bad considering how big (relatively) the data is. I am curious how much faster would it be with async insertion.

Any plan in the future to implement an async API?

[nirinchev]

Are you doing it with a single transaction or a transaction per each insertion? You should definitely do the entire insertion with a single transaction for optimal results. Additionally, have you measured what takes time - the looping or committing the transaction itself? An async API will only affect committing the transaction, not looping through the tables and inserting the rows (as that is regular js code unrelated to realm). Finally, an async API will not make committing the transaction faster - instead, it will avoid blocking the thread while waiting for the commit to happen.

[markhomoki]

Thanks for your comment. After your response, I was doing more debugging and measuring. Looping through those rows takes 7ms. Inserting them is also quite fast ~ 5-6s. The issue seems like reading the file which of course is not related to this library and to reset the schema. So do you have any idea how can I make the delete process faster? Technically before each sync, I'd like to flush the whole DB.

1. Authenticate and receive a zip file containing several .json files for each table (this take 2-3 seconds)
2. Unzipping on a device and getting a list of files from that folder (this is around 0.5s)
3. Looping through the files(schemas) and read each of them with React Native FS (this is around 17s ~ still trying to make this process faster)
4. open a write transaction and clear that schema (this is faster than doing realm.deletaAll() before the initial for loop) (with getting all object and deleting each schema takes 20-25s if I do realm.deletaAll() takes 40s)
5. inside the write transaction loop through all the rows and insert into the DB (this is around 5-6s)

files.map((file) => {
    if (!file.isFile()) {
        return false;
    }

    this.parseFile(file);
});

async parseFile(file) {
    const fileName = file.name.replace('.json', '');
    const data = await this.readFile(file);
    const dataLength = data.length;
    let i = 0;

    if (dataLength > 0) {
        // // clear model before inserting new data
        const allData = realm.objects(fileName);
        realm.write(() => realm.delete(allData));
        // // insert new data
        realm.write(() => {
            for (; i < dataLength; i++) {
                realm.create(fileName, data[i]);
            }
        });
    }
}

[fealebenpae]

You could always just delete the realm file (you can find out where it is with realm.path), but be sure to call realm.close() before you try deleting. Then just open a new instance and write the JSON data.

[acrazing]

For debugging, 1 second may not be so stand out, but in production, 1 second or event 1millisecond is important for user interface, because JavaScript is running in a single thread, which is the reason why all the API about I/O is asynchronous in JavaScript, you know that. If use blocking I/O, means if we use 1 second to query 1 row in 1 table, we need 10 seconds to query 10 rows in 10 different tables, and we can do nothing else when we query the rows.

[nirinchev]

Where do you get this 1 second from? In your application, does it take 1 second to query a single row in a table?

[acrazing]

@nirinchev For reading, 1 millisecond is enough, but if we are writing one row, all other actions will be blocked.
Do you think asynchronous API is unnecessary for this project?

[nirinchev]

For 99% of the use cases - yes. The synchronous API should be super fast and shouldn't affect the UI. If that is not the case with your app, we'd be happy to take a closer look, but providing async API just for the sake of having it is not worth it in my opinion.

[markhomoki]

Thank you guys for your help. Now I could increase the performance a bit. Now it takes 20seconds less than before thanks to deleting the Realm file and then create a new instance.

I had to update the Realm config file. The only issue with this that I have to call getRealmInstance every time I want to access to the instance after the resetRealm function called because import loses the instance. Thanks again and sorry for messing up this issue, as at the end it is not really related to the async API.

const createRealm = () => new Realm(realmConfig);
let realm = createRealm();

const resetRealm = () => {
    realm.close();
    Realm.deleteFile(realmConfig);
    realm = createRealm();
};

const getRealmInstance = () => realm;
const realmInstance = getRealmInstance();

export { getRealmInstance, resetRealm };
export default realmInstance;

[Louies89]

Hi,
I am very new to Realm, and unable to find solution in which way, i should use realm to do operations non-blockingly. I am trying to implement offline first mobile application using React native and Realm.

In my case I am using sockets, so data from server can come at anytime and shall be sent to server anytime.

My schema models are explained here

Can anyone suggest how to do the operations with a small example.

Thank you

[kneth]

@Louies89 Once your Realm is opened, using a listener might be a good idea: https://realm.io/docs/javascript/latest/#notifications

[Louies89]

@kneth
Thank you for the reply..
I have gone through the hole documentation, and I understand the listening part. But My concern is regarding, services I want to implement on my schema and to just export those to anywhere and directly call.

As there is no complete example i found anywhere, I am confused, how to use Realm.Open() and do async operations. You can check the way i tried to implement and got errors in below link:
https://stackoverflow.com/questions/48756313/how-to-open-realmjavascript-object-asynchronously-and-use-it-with-services

If you will suggest some changes in above code I am using, it will be great.

Thank you :)

[kneth]

Realm.open is opening a Realm asynchronously which is particular useful if your app is syncing to a Realm Object Server or you have many changes to your schema (which triggers a migrate of the Realm file). To begin with, I would use new Realm().

Our example is using new Realm instead of Realm.open: https://github.com/realm/realm-js/tree/master/examples/ReactExample. We would like to rewrite it but we have not prioritized it yet.

[Louies89]

I have to sync my back-end MongoDb data to local realm, so I wanted to use it.

As new Realm() is a synchronous operation it will cause problem for me as my data is coming asynchronously.

It will be a great help if some example can be shown.

And another thing is: Realm.Open() is just a promise or it is a transaction?

[kneth]

It's a promise. The first time called, it has to write a few things (create a database for the schema).

We don't a full example, currently only https://realm.io/docs/javascript/latest/#introduction and https://realm.io/docs/javascript/latest/#realms. I guess you will have to query your MongoDB within then().

[Louies89]

If I am using below code to quarry a schema:

Realm.open(dbOptions).then(realm => {
try {
let info=realm.objectForPrimaryKey('RealmSchema1', '123');
return info.name;
} catch(e) {
console.log(e);
}
})
.catch(e => {
console.log(e);
});

The return info.name is not working. So how to resolve and reject using Realm.open() ?

[kneth]

@Louies89 Please create a separate issue - and fill in all the information in the template.

[Louies89]

issue #1662 is created newly for Realm JavaScript Realm.Open() not supporting 'return' statement.

[RealmBot]

➤ Unito Sync Bot commented:

Transition made by Unito

[FansPro]

@nirinchev For reading, 1 millisecond is enough, but if we are writing one row, all other actions will be blocked.
Do you think asynchronous API is unnecessary for this project?
Did you solve this, i'm use realm to do an chat function too