Orleans: MongoDb Membership Provider

[laredoza]

Hi,

My current project is using a replica set MongoDb database. My question is, is their a good reason not to use MongoDb for the Cluster Management?

After reading http://dotnet.github.io/orleans/Documentation/Runtime-Implementation-Details/Cluster-Management.html, it seems that I only need to implement IMembershipTable. Is this correct?

Could I then base my Membership table collection on the MembershipEntry properties?

Where would I put this? I see there is a Mongo Storage Provider under samples, should I do something similar? I guess my question is how do I get involved with this?

My last question is, besides the Storage Provider and the Membership Provider, is there anything else that requires a Mongo implementation?

[sergeybykov]

My current project is using a replica set MongoDb database. My question is, is their a good reason not to use MongoDb for the Cluster Management?

I see no reason for not doing it if you already have MongoDb deployed and used anyway

After reading http://dotnet.github.io/orleans/Documentation/Runtime-Implementation-Details/Cluster-Management.html, it seems that I only need to implement IMembershipTable. Is this correct?

Yes that's the minimum, coupled with IGatewayListProvider.

My last question is, besides the Storage Provider and the Membership Provider, is there anything else that requires a Mongo implementation?

Other "system store" pieces are IReminderTable for reminders, IStatisticsPublisher for runtime statistics, and ISiloMetricsDataPublisher/IClientMetricsDataPublisher for silo/client metrics. They are all optional, but needed if you want to use those features.

Could I then base my Membership table collection on the MembershipEntry properties?

I'm not sure I understand the question. Could you elaborate please.

Where would I put this? I see there is a Mongo Storage Provider under samples, should I do something similar? I guess my question is how do I get involved with this?

We use https://github.com/orleanscontrib as a place for such extensions. Some of them later may get promoted to the main repo. Ask @richorama - he'll be happy to create a repo there for you. And I suspect people in the community might want to join your effort.

[richorama]

Happy to add you as an admin on OrleansContrib. I'm on vacation at the moment, I'm back next week...

Sent from my iPhone

On 22 Sep 2016, at 01:29, Sergey Bykov [email protected] wrote:

My current project is using a replica set MongoDb database. My question is, is their a good reason not to use MongoDb for the Cluster Management?

I see no reason for not doing it if you already have MongoDb deployed and used anyway

After reading http://dotnet.github.io/orleans/Documentation/Runtime-Implementation-Details/Cluster-Management.html, it seems that I only need to implement IMembershipTable. Is this correct?

Yes that's the minimum, coupled with IGatewayListProvider.

My last question is, besides the Storage Provider and the Membership Provider, is there anything else that requires a Mongo implementation?

Other "system store" pieces are IReminderTable for reminders, IStatisticsPublisher for runtime statistics, and ISiloMetricsDataPublisher/IClientMetricsDataPublisher for silo/client metrics. They are all optional, but needed if you want to use those features.

Could I then base my Membership table collection on the MembershipEntry properties?

I'm not sure I understand the question. Could you elaborate please.

Where would I put this? I see there is a Mongo Storage Provider under samples, should I do something similar? I guess my question is how do I get involved with this?

We use https://github.com/orleanscontrib as a place for such extensions. Some of them later may get promoted to the main repo. Ask @richorama - he'll be happy to create a repo there for you. And I suspect people in the community might want to join your effort.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[galvesribeiro]

@laredoza I'm back, so feel free to ping me on Gitter and I can add you to OrleansContrib and help you with the provider if you want. I recently did worked on some providers so its fresh in my mind.

[ashkan-saeedi-mazdeh]

@laredoza I don't want to sound discouraging but if you are using Orleans to have a highly scalable and available solution, I recommend moving away from MongoDB in general.
From MongoDB's documentation

A replica set is a group of mongod instances that maintain the same data set. A replica set contains several data bearing nodes and optionally one arbiter node. Of the data bearing nodes, one and only one member is deemed the primary node, while the other nodes are deemed secondary nodes.

The primary node receives all write operations. A replica set can have only one primary capable of confirming writes with { w: "majority" } write concern; although in some circumstances, another mongod instance may transiently believe itself to also be primary. [1] The primary records all changes to its data sets in its operation log, i.e. oplog. For more information on primary node operation, see Replica Set Primary.

If you want a scalable document store, look at other options like CouchBase , Riak or ... which don't use a primary/secondary or Master/Slave architecture. I heard the scenario of global locks are a bit improved in MongoDB but still if you go large scale , you'll not have not that great days with this popular thing. Google for MongoDB issues , MongoDB scalability issues and ... to get more credible examples than an unknown developer's writing on GH.

[laredoza]

I'd just like to thank everybody for their replies

@sergeybykov

With regards to:

Could I then base my Membership table collection on the MembershipEntry properties?

[Serializable]
public class MembershipEntry
{
public SiloAddress SiloAddress { get; set; }

    public SiloStatus Status { get; set; }

    public List<Tuple<SiloAddress, DateTime>> SuspectTimes { get; set; }

    public int ProxyPort { get; set; }

    public string HostName { get; set; }

........

I was thinking that this would be the basis for the information to be saved in the database so that an implementation of IMembershipTable would work.

@galvesribeiro
I would appreciate the help. I'll get in touch with you on Gitter. I will probably be looking at this project next week. I don't have much experience with Git / Github, so a little patience might be required. I'll go through a few youtube videos tomorrow so that I can get up to speed.

@ashkan-saeedi-mazdeh
We have severe resource constraints at the moment, hence the choice of a replication set. We will probably be moving to a Sharded Cluster once the resources become available. This will allow specifying different read / write servers. At the moment the replication set will have to do. NoSql was a tough sell, so Mongodb's popularity helped a lot in getting the go ahead. So far the performance has been great.

[sergeybykov]

I was thinking that this would be the basis for the information to be saved in the database so that an implementation of IMembershipTable would work.

You will have to store all the properties of MembershipEntry in the database except for the three optional ones.

[galvesribeiro]

@laredoza perfect! Just ping me there when you need. 👍

[galvesribeiro]

@laredoza I created the project on OrleansContrib and gave you admin access on it. You should receive an email from GH with the invite to the organization.

[laredoza]

@galvesribeiro Thanks

[laredoza]

I've based my MembershipTable & ReminderTable's on the sql versions in Orleans. Implementing their unit tests has caused a few questions.

With respect to the MembershipTable:

I'm storing the data in a OrleansMembership collection. The date fields are stored as UTC values. At the moment I'm returning the values as their UTC values when I query it? Should I be returning the dates in their local format? If I look at the base test it uses the following to create the test dates:

private static DateTime GetUtcNowWithSecondsResolution()
        {
            var now = DateTime.UtcNow;
            //return new DateTime(now.Year, now.Month, now.Day, now.Hour, now.Minute, now.Second);
            return now;
        }

If I uncomment the second line the MembershipTable_MongoDB_UpdateRow() test passes. This indicates to me that I should be returning the date values as local time. Just want to confirm.

With respect to the ReminderTable:

The original test in the Orleans library adds a reminder with a 10sec time. Currently I have to use a minimum value of 1 min otherwise I receive an exception. This makes the tests take forever. Is there a way that I can set this time to 10 seconds as in the original tests?

[sergeybykov]

This indicates to me that I should be returning the date values as local time. Just want to confirm.

Doesn't look like that to me. Seems like just a "rounding" of the UTC time to the second. Not sure why.

The original test in the Orleans library adds a reminder with a 10sec time.

Which test?

[laredoza]

This indicates to me that I should be returning the date values as local time. Just want to confirm.
Doesn't look like that to me. Seems like just a "rounding" of the UTC time to the second. Not sure why.

Thanks

The original test in the Orleans library adds a reminder with a 10sec time.

Which test?

The class ReminderTests_Base's method Test_Reminders_Basic_StopByRef() calls await grain.StartReminder(DR).

This belongs to ReminderTestGrain2

public async Task<IGrainReminder> StartReminder(string reminderName, TimeSpan? p = null, bool validate = false)
        {
            TimeSpan usePeriod = p ?? period;
            logger.Info("Starting reminder {0}.", reminderName);
            IGrainReminder r = null;
            if (validate)
                r = await RegisterOrUpdateReminder(reminderName, usePeriod - TimeSpan.FromSeconds(2), usePeriod);
            else
                r = await RuntimeClient.Current.RegisterOrUpdateReminder(reminderName, usePeriod - TimeSpan.FromSeconds(2), usePeriod);

            allReminders[reminderName] = r;
            sequence[reminderName] = 0;

            string fileName = GetFileName(reminderName);
            File.Delete(fileName); // if successfully started, then remove any old data
            logger.Info("Started reminder {0}", r);
            return r;
        }

and period is set by

public static TimeSpan GetDefaultPeriod(Logger log)
        {
            int period = 10; // Seconds

            ClusterConfiguration config = new ClusterConfiguration();
            config.LoadFromFile("ClientConfigurationForTesting.xml");
            if (config.Globals.UseAzureSystemStore)
            {
                period = 12; // azure operations take more time ... so we use a larger period
            }
            else if (config.Globals.UseSqlSystemStore)
            {
                period = 12; // SQL operations are quite fast ... so we use a shorter period
            }
            var reminderPeriod = TimeSpan.FromSeconds(period);
            log.Info("Using reminder period of {0} for ReminderServiceType={1} in ReminderTestGrain", reminderPeriod, config.Globals.ReminderServiceType);
            return reminderPeriod;
        }

I had to set period to 60s otherwise I got the above mentioned error.

I found the constant, it is Constants.MinReminderPeriod. It's an internal class so I've used reflection to set the value in my UnitTests. I've added the following to StartReminder

var ass = Assembly.GetAssembly(typeof(ClientConfiguration));
var type = ass.GetType("Orleans.Runtime.Constants");
var field = type.GetField("MinReminderPeriod");
field.SetValue(null, TimeSpan.FromSeconds(10));

It works, not sure if there is a better way to do this

[sergeybykov]

The test grain intentionally bypasses the min period check by calling the reminder service directly RuntimeClient.Current.RegisterOrUpdateReminder() when validate==false. Otherwise, it makes a normal call to RegisterOrUpdateReminder() which requires period to be >= 1 minute.

[laredoza]

Thanks, I will implement.

[laredoza]

Is it okay if I close this issue? I've made my final changes to the MongoStatisticsPublisher.

[ReubenBond]

@laredoza go ahead :)

[laredoza]

Thanks for all the help guys