Mongodb-memory-server: Does it work with mongodb 4?

[YuriGor]

Latest working version of mongo I tried is 3.6.3
When I try latest or 4.0.0/1/2/3 - it downloads binaries and then silently exits the process

(K)Ubuntu 18.04
Node v10.7.0
mongodb-memory-server 2.4.3

[AJRdev]

@YuriGor Did you specified your mongodb version with the env variable MONGOMS_VERSION ?

[YuriGor]

No, I used default constructor parameter like this:

var mongod = new MongodbMemoryServer.default({
    instance: {
      dbName: 'slug',
    },
    binary: {
      version: '3.6.3',
    },
  });

here is my test script:
https://gist.github.com/YuriGor/cc8114a0d0b1ad151eaf58baab24aa3a
(actually it was created for testing possible bug in mongoose)

[AJRdev]

In your test script, didn't you forgot to add a await keyword to this line ?

mongoose.connect(
    mongoUri,
    { useNewUrlParser: true },
  );

If you don't wait this Promise to resolve, you will not be able to do a schema.create() after I think and it might explain why the process exits.

[YuriGor]

connect is synchronous method

It has a callback function parameter to handle errors, and I added one:
https://gist.github.com/YuriGor/cc8114a0d0b1ad151eaf58baab24aa3a#file-pre-update-js-L26
but with no effect, it still works fine with 3.6 3 and stops silently with 4.0.*

I also tried to add 'await' also with no success.

[AJRdev]

@YuriGor Where do you see that is a synchronous method ? In the mongoose documentation, it is specified that is a Promise that is returned so it suggests that is a asynchronous operation.
But the docs also specify that if you don't wait the connect method to resolve it's Promise, you will be able to define mongoose schemas but the first operation that you will do on the database will hang on until mongoose is connected to the database and will not throw any error :
https://mongoosejs.com/docs/connections.html

Mongoose lets you start using your models immediately, without waiting for mongoose to establish a connection to MongoDB.

mongoose.connect('mongodb://localhost:27017/myapp');
var MyModel = mongoose.model('Test', new Schema({ name: String }));
// Works
MyModel.findOne(function(error, result) { /* ... */ });
That's because mongoose buffers model function calls internally. This buffering is convenient, but also a common source of confusion. Mongoose will not throw any errors by default if you use a model without connecting.

So there is certainly a problem in the connection with a 4.0 mongodb binary and you couldn't catch it.
And when you say it stops silently, where exactly in the code does it stop ?

[YuriGor]

it stop at line
const mongoUri = await mongod.getConnectionString();
so mongoose even doesn't try to connect

[AJRdev]

I tried your code with the version 4.0.3 of mongodb and it worked, I had this output :

pre-update defined with default(?) options/ 66.8mb)
pre-update defined by regex with default(?) options
mondo connected
I'll be updated soon
-----------------
default preupdate!
default this looks good
-----------------
default regex preupdate!
default regex this looks good
(node:79695) DeprecationWarning: collection.update is deprecated. Use updateOne, updateMany, or bulkWrite instead.
I was updated!

Edit : I just saw you were on ubuntu, I'm on mac so I will give it a try with a ubuntu docker image !

[YuriGor]

default regex preupdate!
default regex this looks good

oh wow, regex hook works too! for me it doesn't
and what do you see with v3.6.3?

I have latest MongoDB already installed on my machine, can it conflict with one started by mongodb-memory-server?
May be this is why older version still works, but not latest?

[AJRdev]

I have the same output with version 3.6.3 ! But like I specified before I'm running on mac os, I will try it on a ubuntu distrib.

[YuriGor]

I tried to remove mngodb instalation, rebooted, but has same issue.
I will try to install mongodb again and use it in this test without mongodb-memory-server's help, may be it's a mongodb issue

[YuriGor]

It works fine directly with mongo 4.0.3
but regex hooks still don't work in mongoose (wtf?)
What is your node and mongoose versions?

[YuriGor]

Ok, i had an old Mongoose version, that's why regex didn't work, but there is still a problem with mongodb-memory-server, it can't run mongodb 4.0.3

[YuriGor]

Here is isolated test script for this issue
https://gist.github.com/YuriGor/035fca75e53fd14204a3a4d44eee2f6a
I'll modify that one above to remove mongodb-memory-server and work with mongodb directly

[floric]

I had the same issue unexpectedly. My tests kept failing as I didn't specify the Mongodb version to 3.6.x. Apparently it used the 4.x version which never passed client.getConnectionString().

[nodkz]

@YuriGor @floric update to the latest package version and try to run following code:

const MongodbMemoryServer = require('mongodb-memory-server').default;

async function main(v) {
  try {
    console.log(v);
    console.log('--> b4 new MongodbMemoryServer.default..');
    var config = {
      instance: {
        dbName: 'run',
      },
      binary: {
        version: v,
      },
      debug: true,
    };
    const mongod = new MongodbMemoryServer(config);
    console.log('--> b4 mongod.getConnectionString()..');
    const mongoUri = await mongod.getConnectionString();
    console.log(
      `[!] If you see this, then mongodb-memory-server works with mongo ${config.binary.version}`
    );
    await mongod.stop();
    console.log(`stop ${config.binary.version}`);
  } catch (err) {
    console.error(`Something wrong with ${config.binary.version}`, err);
  }
}

main('4.0.0')
  .then(() => {
    return main('4.0.1');
  })
  .then(() => {
    return main('4.0.2');
  })
  .then(() => {
    return main('4.0.3');
  })
  .catch(err => {
    console.log('oh noes');
  });

It will output debug information which helps to resolve your issue.

[nodkz]

I confirm that it works with mongodb v4 on macOS, also it works on ubuntu 14.04 (travis CI).

[floric]

WIll have a look later. FYI currently, I don't need Mongodb 4.x. It just happened by accident as the version was bumped. But I'll stick to Db 3.6 for now :) Thanks anyway.

[YuriGor]

I ran your script and found error

version `CURL_OPENSSL_3' not found (required by /home/gor/.mongodb-binaries/4.0.0/mongod)

4.0.0
--> b4 new MongodbMemoryServer.default..
Autostarting MongoDB instance...
--> b4 mongod.getConnectionString()..
Thu, 25 Oct 2018 16:55:34 GMT Mongo[41659] Starting MongoDB instance with following options: {"port":41659,"dbName":"run","uri":"mongodb://127.0.0.1:41659/run","storageEngine":"ephemeralForTest","dbPath":"/tmp/mongo-mem-15929Tdoexfi6klUy"}
Thu, 25 Oct 2018 16:55:34 GMT Mongo[41659] MongoBinary options: {"downloadDir":"/home/gor/.mongodb-binaries","platform":"linux","arch":"x64","version":"4.0.0"}
Thu, 25 Oct 2018 16:55:34 GMT Mongo[41659] MongoBinary: Mongod binary path: /home/gor/.mongodb-binaries/4.0.0/mongod
Thu, 25 Oct 2018 16:55:34 GMT Mongo[41659] MongoBinary: Download lock removed
Thu, 25 Oct 2018 16:55:34 GMT Mongo[41659] STDERR: /home/gor/.mongodb-binaries/4.0.0/mongod: /usr/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by /home/gor/.mongodb-binaries/4.0.0/mongod)

Thu, 25 Oct 2018 16:55:34 GMT Mongo[41659] CLOSE: 1
[Finished in 2.3s]

so I tried to
sudo apt-get install libcurl3
and it helped, now my tests work with version 4.0.3

But how did standalone installed MongoDB work without installed libcurl3?
Does it have this lib bundled? Then why binaries downloaded by mongodb-memory-server do not?
May be we need to show this error without debug flag too?

[nodkz]

@YuriGor can you take care about it and make PR?

We cannot install libcurl3 but may provide proper error message.
It can be started here https://github.com/nodkz/mongodb-memory-server/blob/132df61731690f0e91d9b8378c1f49c6b85ca1ee/src/util/MongoInstance.js#L158

[YuriGor]

Ok, but not very soon.

[YuriGor]

I just noticed that binaries archive files downloaded into .mongodb-binaries folder has names like this:
mongodb-linux-x86_64-ubuntu1404-4.0.3.tgz
so this is version for Ubuntu 14.04, where libcurl3 was in use.

I manually downloaded mongodb-linux-x86_64-ubuntu1804-4.0.3.tgz from https://www.mongodb.com/download-center/community and replaced mongod in .mongodb-binaries/4.0.3/ folder.
Now mongodb-memory-server runs Mongo v4.0.3 with default for Ubuntu 18.04 libcurl4 library installed, no need to instal libcurl3.
This is important, because libcurl3 and libcurl4 are mutually exclusive and in case libcurl3 will be installed instead of libcurl4, I expect some compatibility issues with other soft aimed to Ubuntu 18.04.

So mongodb-memory-server needs to choose correctly which binaries to download considering current OS version.

[YuriGor]

I believe this versions list should be updated:
https://github.com/nodkz/mongodb-memory-server/blob/132df61731690f0e91d9b8378c1f49c6b85ca1ee/src/util/MongoBinaryDownloadUrl.js#L191
lates Ubuntu version listed here is 16.04, so in case of 18.04 default 14.04 used instead.

[nodkz]

@YuriGor great catch!!!

[YuriGor]

Issue solved with #96 so I close it.
Thank you for help!