Realm-js: [iOS] EXC_BAD_ACCESS

[FlaviooLima]

My app crash every time I try to do this and only this query to the realm database.
It happens only in iOS and in Debug Mode.
Release mode works as expected.

PS:
To clarify when I say debug Mode, I don't mean the Debug Mode "Debug JS Remotely", what I mean is Application Scheme Build Configuration in Debug.

Query:
realm.objects('playersInfo').filtered(queryString + " OR " + queryString2, [200, 500], this.props.onLoan, userInfo.SystemVisible).sorted('updated', true);

App Crash error:

Error: com.facebook.react.JavaScript (10): EXC_BAD_ACCESS (code=2, address=0x7000080b1f88)

(lldb) register read
General Purpose Registers:
rbx = 0x0000000128600000
rbp = 0x00007000014edf10
rsp = 0x00007000014ed620
r12 = 0x0000000000000004
r13 = 0x00007000014ee210
r14 = 0x00007000014ee068
r15 = 0x0000000000000004
rip = 0x000000010c0d23c9 MyApplicationName`OpaqueJSValue* realm::js::ResultsClass::create_filtered >(OpaqueJSContext const*, realm::js::Results const&, realm::js::Arguments) + 505 at js_results.hpp:190
13 registers were unavailable.

Goals

To retrieve the results.

Expected Results

To retrieve the results.

Actual Results

App crash.

Error: com.facebook.react.JavaScript (10): EXC_BAD_ACCESS (code=2, address=0x7000080b1f88)

Steps to Reproduce

Maybe have a query string really really big.
Still trying to reproduce.

Version of Realm and Tooling

"realm": "^2.7.2"
"react-native": "0.52.1",
Client OS & Version: High Sierra

[kneth]

How long is your query string? The communication between the app and the debugger is implemented using HTTP requests, so there might be an upper limit.

[FlaviooLima]

@kneth Hi, thanks for the response.
29259 character.
But the strangest thing is why in Release mode it works :/
Only debug mode crash.
To clarify when I say debug Mode, I don't mean the Debug Mode "Debug JS Remotely", what I mean is Application Scheme Build Configuration in Debug.

[kneth]

29K shouldn't be much.

@ironage Do you know of limitations in the query parser which could explain this?

[FlaviooLima]

@kneth @ironage
Is there any way I can give you more information?

[ironage]

@FlaviooLima I'd be interested to know more about your query, can you describe the structure of queryString and queryString2 more? If it doesn't contain any private information, could you send the whole query string? If you are able to reproduce this with a minimal query that would also be extremely valuable.

[FlaviooLima]

@ironage @kneth
From what I understand it doesn't matter if:

• it's inside "()" or not.
• if you use "OR" or "AND".
• If the field that you are looking for exists. (Change id_server and error still occurs).
  -It's not the number of character but the number of conditions in the string.

I develop a simple string creation that replicates this issue:
Can you test it if it happens to you?

Test 1 (Doesn't Work):

var testQuery = "";
const totalLength = 1088;

for (let index = 0; index < totalLength; index++) {
   testQuery += (index >= (totalLength - 1)) ? " id_server == 54875" : " id_server == 54875 OR";
}

return realm.objects('myScheme').filtered(testQuery);

Test 2 (Doesn't Work):

var testQuery = "";
const totalLength = 1088;

for (let index = 0; index < totalLength; index++) {
   testQuery += (index >= (totalLength - 1)) ? " id == 5" : " id == 5 OR";
}

return realm.objects('myScheme').filtered(testQuery);

Test 3 (Work):

var testQuery = "";
const totalLength = 1087;

for (let index = 0; index < totalLength; index++) {
   testQuery += (index >= (totalLength - 1)) ? " id_server == 54875" : " id_server == 54875 OR";
}

return realm.objects('myScheme').filtered(testQuery);

Test 4 (Work):

var testQuery = "";
const totalLength = 1087;

for (let index = 0; index < totalLength; index++) {
   testQuery += (index >= (totalLength - 1)) ? " id == 5" : " id == 5 OR";
}

return realm.objects('myScheme').filtered(testQuery);

[kneth]

Thanks for the simple tests. Looks like 1087 is a magic number and 1088 isn't 😢. We stil use it in our quest to reproduce it - and probably write a test for Realm Core where the query engine lives.

[FlaviooLima]

@kneth I can confirm that 1087 is not the magic number. :(
My colleague tried it with 1088 and it worked, but it failed with 2000;
Different machine but same scheme configuration.

[kneth]

I tried to reproduce it in a core test but it passes (with 1088). I am certain that you don't hit the max size of a string: https://github.com/nodejs/node/blob/ec02b811a8a5c999bab4de312be2d732b7d9d50b/deps/v8/include/v8.h#L2095

[FlaviooLima]

@kneth Check my last comment. The number is not equal to everyone. :/
Can you try an absurd number like 5000 or even 10000.
I'm trying to apply the test to the core that you made and check a few values to see if it fails.

Update:
Testing with commands:
cmake --build . && ctest

and with: (from test code that you push)

const size_t max_terms = 10190;
and no errors:

    Start 1: RealmTests
1/3 Test #1: RealmTests .......................   Passed   80.51 sec
    Start 2: RealmBenchmarkCommonTasks
2/3 Test #2: RealmBenchmarkCommonTasks ........   Passed  361.34 sec
    Start 3: RealmBenchmarkCrud
3/3 Test #3: RealmBenchmarkCrud ...............   Passed  1171.33 sec

Right now I feel that I'm chasing a ghost.

[ironage]

Hi @FlaviooLima thanks for the info, I can reproduce this by using 10000. It is a stack overflow. In release mode, many of the functions would be inlined and optimised out which explains why you only see it in debug mode. We can look at fixing this but it may take some time. Are you able to rework your design to reduce the size of the query? If you are able to do that, it will also reduce the execution time of parsing the query.

[FlaviooLima]

Hi @ironage thanks for the response :)
To be honest this was the optimized way for performance xD
But I believe that I can achieve this in other ways :)
Guess its time to start reading the books again, probably I'm doing this wrong!!! :)

[ironage]

@FlaviooLima Thanks for reporting this and for helping us reproduce it! Hopefully we can make a fix soon.

If you have top level "AND" conditions, you could reduce the query size by filtering multiple times so instead of filtered(queryString + " AND " + queryString2, args) you could write filtered(queryString, args).filtered(queryString2, args). Hope that helps until we can get a fix out. Good luck with rewriting the query.

[FlaviooLima]

@ironage Thanks for the response =D
But unfortunately, I'm already doing that xD.
And the part that breaks the code is really "queryString2" here there are only "OR".
I already found a way to work around this but it's such a nasty way that I'm trying to see if any other idea comes to mind xD
@kneth @ironage thank you very much, hope to see this fix in the future and keep up the awesome work =D 🚀 🌕 🌕

[jedelbo]

Changing back to @ironage after having understood the problem. No easy fixes here.

[mjm918]

I faced the same issue. In my case, the query string length was 24376. I could not find any solution but I found a workaround. I split the query string into half and filtered the data twice. I know it's not a good practice but serve the purpose.

      let query = customers.map((id) => 'field_id == ' + id).join(' OR ');

        var middle = Math.floor(query.length / 2);
        var before = query.lastIndexOf('OR', middle);
        var after = query.indexOf('OR', middle + 1);

        if (middle - before < after - middle) {
            middle = before;
        } else {
            middle = after;
        }

        var q1 = query.substr(0, middle);
        var q2 = query.substr(middle + 2);
        q1 = q1.trim();
        q2 = q2.trim();

Let me know if you guys found any solution

[kneth]

@mjm918 Thanks for details.

@ironage @jedelbo Can it be a limitation in the query parser?

[ironage]

@kneth we need to take some time and rewrite a couple parser rules to not be recursive. It should be possible, but we just need to prioritise it.

[kaaoo]

Hello! Any updates on this?:)

[nghiant96]

Hello! Any updates on this?

[jedelbo]

We have just re-implemented the query parser and the new parser should not suffer from this problem.