Meteor-feature-requests: Provide access to the `MongoClient` for transactions & change streams

[klaussner]

Using transactions and deployment-level change streams in MongoDB 4.0 (meteor/meteor#10058) requires a reference to the MongoClient instance (to call MongoClient#startSession and MongoClient#watch). However, we can currently get only the raw collection and the raw database from a Mongo.Collection. An easy way to provide access to the client would be to add a new rawClient function, similar to the existing functions:

• Mongo.Collection#rawCollection
• Mongo.Collection#rawDatabase
• New: Mongo.Collection#rawClient

However, the client isn't really tied to a collection, so it can be a little inconvenient to access it via a Mongo.Collection (the same applies to Mongo.Collection#rawDatabase). Maybe we could add these two functions to Mongo instead:

• Mongo#rawDatabase
• Mongo#rawClient

[StorytellerCZ]

Or we could add the startSession and related functionality to Mongo package itself so that there is no need to call rawClient.

[smeijer]

If they are not bound to the Collection, why not have something like:

import { Database, Client, startSession } from 'meteor/mongo';

Or if we really want to keep clear that we are using raw methods, something like:

import { raw: { Database, Client, startSession } } from 'meteor/mongo';

I should say that I'm not in favor of keeping the raw namespacing myself. I'm using more and more raw methods, now there is async/await support. I'm moving all database communication into apollo resolvers. The db is there injected trough the context, and all collections are rawCollections. Mark the method as being async and it's working just fine.

async blog(_, { id }, { db }) {
  // db.blogs = new Collection('Blogs').rawCollection(); 
  const blog = await db.blogs.findOne({ _id: id }); 
  return blog;
}

I guess it's all thanks to the meteor promise implementation, which wraps all Promises / async await into Fibers.

[klaussner]

I think we shouldn't expose specific client functions like startSession in the Mongo API because it wouldn't be immediately clear if a function is part of Meteor's API or a raw driver function (which might be subject to breaking changes if the driver is updated :warning:). Exporting a raw object would work but I don't see the benefit compared to a Mongo.rawClient function. :smile: For example:

import { raw } from "meteor/mongo";
const session = raw.startSession(); // or `raw.Client.startSession`?

// versus

import { Mongo } from "meteor/mongo";
const session = Mongo.rawClient().startSession();

An alternative to exposing the client would be to implement support for transactions and change streams in the Mongo API but I don't think that it's worth the maintenance overhead. Adding a rawClient function would be a minimal change that allows package authors to build abstractions for these new features—and features that will be added to the driver in the future.

[msavin]

[phone]

[radekmie]

Well, _technically_, it is possible:

import {MongoInternals} from 'meteor/mongo';

const {client, db} = MongoInternals.defaultRemoteCollectionDriver().mongo;

Although, I agree for an additional API.

[mitar]

Just to document this somewhere:

• I think that it might be also useful if we could pass session to existing Meteor MongoDB operations inside options argument.
• Probably in oplog tailing we should filter all messages which are inside transactions, for pub/sub to not process those and send them to clients. (I am guessing this is necessary.)
• On the server, we could provide some nice blocking utility function to run a series of commands inside a transaction, like collection.insideTransaction(() => {...}).

[klaussner]

I think that it might be also useful if we could pass session to existing Meteor MongoDB operations inside options argument.

Do you mean that collection functions in the meteor/mongo API should get a new session option or that we should use sessions internally in combination with an insideTransaction function?

[mitar]

Do you mean that collection functions in the meteor/mongo API should get a new session option

Yes. That should be added.

or that we should use sessions internally in combination with an insideTransaction function?

That could be done as well, as a sugar, on a server inside a fiber.

[mitar]

That could be done as well, as a sugar, on a server inside a fiber.

And the reason for this is that it is pretty tricky if you forget session. If you do, operation will just go through, but not inside a transaction. Same for reading, it might happen that you go and findOne data outside of transaction by accident, breaking isolation. Now imagine that you are also calling into other functions which might not even know that they are inside a transaction operating at a moment.

So yea, this sugar is probably pretty critical.

[klaussner]

How would you create session objects? With the native startSession driver function or a new API?

[mexin]

We've been trying to incorporate transactions to our meteor project and we have found the following in order to use them, hope it helps someone as there isn't a solid milestone for this to be added into Meteor

To initialize the session for the transaction we use the following

const { client } = MongoInternals.defaultRemoteCollectionDriver().mongo;
    const session = client.startSession();
    session.startTransaction();

If we have a defined Collection using regular new Mongo.Collection()
then we can use const RawCollection = Collection.rawCollection() this collections are needed due they allow passing session as a option parameter.

we need to pass session to the rawCollection

const docInsert = await RawCollection.insert({ name: 'Test'}, { session });

If we console log docInsert we get the following

I20190204-19:45:33.498(-6)? { result: { ok: 1, n: 1, opTime: { ts: [Object], t: 1 } },
I20190204-19:45:33.499(-6)?   ops: [ { name: 'Test', _id: 5c58eabd1867114f224b1793 } ],
I20190204-19:45:33.499(-6)?   insertedCount: 1,
I20190204-19:45:33.499(-6)?   insertedIds: { '0': 5c58eabd1867114f224b1793 } }

Also we need to be careful if the doc we are trying to retrieve is on a transaction as if we do any find/findOne without session option it will return null

so we need to use also rawCollection with session on find/findOne

const docFind = await RawCollection.findOne({ _id: docInsert.ops[0]._id }, { session });

I used the returned data from the insert example, hence I get the id from opts array

Also we need to attach _id as _id: Random.id() before inserting, otherwise Mongo driver will use a ObjectId()

Transactions only works on replicate sets, Locally we can use https://www.npmjs.com/package/run-rs to simulate the replica set locally (run with run-rs --mongod)