Parse-server: MongoError: Transaction numbers are only allowed on storage engines that support document-level locking

[funkenstrahlen]

I updated my parse server installation to version 3.8.0. I first got this error in the logs:

MongoError: This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.

This was of course an easy fix and the error went away after adding the option to my connection string.

Now I see another error for queries to the database:

2019-08-27T07:44:31.538322+00:00 app[web.1]: MongoError: Transaction numbers are only allowed on storage engines that support document-level locking
2019-08-27T07:44:31.538354+00:00 app[web.1]: at Connection.<anonymous> (/app/node_modules/mongodb/lib/core/connection/pool.js:466:61)
2019-08-27T07:44:31.538359+00:00 app[web.1]: at emitTwo (events.js:126:13)
2019-08-27T07:44:31.538360+00:00 app[web.1]: at Connection.emit (events.js:214:7)
2019-08-27T07:44:31.538361+00:00 app[web.1]: at processMessage (/app/node_modules/mongodb/lib/core/connection/connection.js:364:10)
2019-08-27T07:44:31.538362+00:00 app[web.1]: at Socket.<anonymous> (/app/node_modules/mongodb/lib/core/connection/connection.js:533:15)
2019-08-27T07:44:31.538363+00:00 app[web.1]: at emitOne (events.js:116:13)
2019-08-27T07:44:31.538364+00:00 app[web.1]: at Socket.emit (events.js:211:7)
2019-08-27T07:44:31.538365+00:00 app[web.1]: at addChunk (_stream_readable.js:263:12)
2019-08-27T07:44:31.538369+00:00 app[web.1]: at readableAddChunk (_stream_readable.js:250:11)
2019-08-27T07:44:31.538370+00:00 app[web.1]: at Socket.Readable.push (_stream_readable.js:208:10)
2019-08-27T07:44:31.538371+00:00 app[web.1]: at TCP.onread (net.js:601:20)

I tried to find information on this issue myself and I think this is caused by my older MongoDB Server version. It currently runs 3.6.11 which does not support this feature.

I have not seen this issue on older releases of parse server. Any other config I need to change to make this work? Or is MongoDB 4 required for parse server now?

• Server

  • parse-server version: 3.8.0
  • Operating System: heroku-18 stack
  • Localhost or remote server? Heroku

• Database

  • MongoDB version: 3.6.11
  • Localhost or remote server? mlab

[davimacedo]

@funkenstrahlen it should work fine for Mongo 3.6 as you can see here. Would you be able to debug and identify which operation is causing the error?

[wfilleman]

I'm getting the same error on my instance as well.

MongoDB 3.6.11
mlab

The error occurs when add a user to my Admin role:

adminRole.getUsers().add(parseUser);
adminRole.save({},{ useMasterKey:true }); // Transaction numbers are only allowed on storage engines that support document-level locking

[davimacedo]

@wfilleman I tried to reproduce the error with this operation but it worked fine for me. Do you mind to write a small js script that could reproduce the error so I can run here and analyze?

[funkenstrahlen]

I just tried to debug the issue. However it is difficult for me because the issue only occurs on production because only my production server actually uses a mlab database cluster. I do not see the issue in my testing environment which runs on a free mlab instance.

Maybe this is the cause why this error does not show up in parse server CI testing? Because it also tests on a simpler MongoDB setup? This is just a guess though.

[wfilleman]

@funkenstrahlen Same here. I run mlab's database cluster in production and that's where I see the issue as well.

[davimacedo]

Do you guys now your mlab production settings? Version, topology, and engine?

We can set mongodb-runner to run in a similar one through env vars. Take a look here. The CI actually run different scenarios as you can see here.

[wfilleman]

@davimacedo I'm not 100% sure where that info is in mlab. There's output from rs.conf() that looks like config data for the production servers. Is there something in that output I can provide?

[funkenstrahlen]

I tried to create a new test case scenario to match my production mlab setup via this PR: https://github.com/parse-community/parse-server/pull/5999

However the travis test build did not find any issues when running through the test suite as you can see in the PR.

My mlab setup is 3.6.11 (MMAPv1). Configuration output looks like this:

{
  "_id": "XXXXXXXXX",
  "version": 3,
  "protocolVersion": 1,
  "members": [
    {
      "_id": 0,
      "host": "XXXXXXXXXXXXXXXXX",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 1,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    },
    {
      "_id": 1,
      "host": "XXXXXXXXXXXXXXXXX",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 1,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    },
    {
      "_id": 10,
      "host": "XXXXXXXXXXXXXXXXX",
      "arbiterOnly": true,
      "buildIndexes": true,
      "hidden": false,
      "priority": 0,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    }
  ],
  "settings": {
    "chainingAllowed": false,
    "heartbeatIntervalMillis": 2000,
    "heartbeatTimeoutSecs": 10,
    "electionTimeoutMillis": 10000,
    "catchUpTimeoutMillis": 60000,
    "catchUpTakeoverDelayMillis": 30000,
    "getLastErrorModes": {},
    "getLastErrorDefaults": {
      "w": 1,
      "wtimeout": 0
    },
    "replicaSetId": {
      "$oid": "XXXXXXXXXXXXXXXX"
    }
  }
}

(I censored the keys with XXXXX)

I assume creating a new test matrix entry like this in the PR should match the setup right?

MONGODB_VERSION=3.6.11 MONGODB_TOPOLOGY=replicaset MONGODB_STORAGE_ENGINE=wiredTiger

MMAPv1 is already used as default when running the tests. I am only unsure about the replicaset configuration as I am not really familiar with its meaning in the travis config. Does it correctly represent my actual setup?

I really hoped to reproduce the error with this test scenario. No luck yet though.

[davimacedo]

I think you should test with:

- MONGODB_VERSION=3.6.11 MONGODB_TOPOLOGY=replicaset

It will make the test to run with the storage engine MMAPv1 which is the default.

I will also try to reproduce it later today.

[funkenstrahlen]

I tested with your recommended configuration in this PR: https://github.com/parse-community/parse-server/pull/6013

- MONGODB_VERSION=3.6.11 MONGODB_TOPOLOGY=replicaset

Unfortunately this also did not reproduce the error.

Maybe it does only occur when retryWrites=false is set on the connection string? I could not find out how to set this parameter in the travis testing environment of parse server though.

[davimacedo]

I also tried and couldn't reproduce. There is no such option to set retryWrites=false via env vars in Travis, but you can create a new test case and use reconfigureServer to set your mongo options. Something similar to this: https://github.com/parse-community/parse-server/blob/master/spec/ReadPreferenceOption.spec.js#L50

[omairvaiyani]

+1 I had this error until I added retryWrites=false to my database uri.

[funkenstrahlen]

So maybe I have a problem with my database uri?

[omairvaiyani]

you could paste it here for us to dissect - take care to remove the hostname and authorisation details like so:
mongodb://xx:[email protected]:33427,xx.com:33427/database-name?replicaSet=rs-xx&ssl=true&retryWrites=false

[funkenstrahlen]

I feel very bad right now because I made an obvious mistake!

I appended ?retryWrites=false to the URI. However this is wrong as there is already the replicaSet parameter and therefore I need to append &retryWrites=false to make it work correctly.

So this is how it looks right now:

[...]?replicaSet=rs-ds00000?retryWrites=false

Working perfectly without any errors.

Thanks for all your help and patience.

[mtrezza]

If I understand correctly, the takeaway here that the storage engine was MMAP1 which does not support retry writes and thus needs retryWrites=false in the connection string.

@davimacedo If this affects users who upgrade to 3.8 it may be worth adding a release note regarding this?

[davimacedo]

I've just taken a closer look in the MongoDB docs to understand the reasons. So let's go:

• In the last release of Parse Server, 3.8, we've upgraded our MongoDB driver to its latest version v3.3.0;
• According to the MongoDB driver v3.3.0 release notes, it is now compatible with the MongoDB v4.2;
• Looking at Retryable Writes documentation, it says that:

The official MongoDB 4.2-compatible drivers enable Retryable Writes by default. Applications upgrading to the 4.2-compatible drivers that require retryable writes may omit the retryWrites=true option. Applications upgrading to the 4.2-compatible drivers that require disabling retryable writes must include retryWrites=false in the connection string.

So, yes. If you are running your Parse Server on top of a MongoDB deployment which does not fit the Retryable Writes Requirements, you will have to add retryWrites=false to your connection string in order to upgrade to Parse Server 3.8.

@mtrezza I think it is worth to mention that in the release notes. Do you want to open a PR with this?

[henrik]

Don't forget to quote the URL if setting this on Heroku. So do:

heroku config:set MONGODB_URI="mongodb://…&retryWrites=false"

with the double quotes. Otherwise the "&" will trip you up.

[asifhajiyev]

Don't forget to quote the URL if setting this on Heroku. So do:

heroku config:set MONGODB_URI="mongodb://…&retryWrites=false"

with the double quotes. Otherwise the "&" will trip you up.

For me ?retryWrites=false is worked (I used ? instead of &) with & it couldn't recognize username and password and with & also using double quotes the entire app didn't worked.