Orleans: Google Cloud Platform (GCP) integration

[galvesribeiro]

This issue is part of #2005 by implementing a minimal set of providers to make Orleans to run well by integrating Google Cloud Platform (GCP).

• [ ] Cloud DataStore Grain State persistence provider
• [ ] Cloud DataStore Membership Provider
• [ ] Cloud DataStore Reminder Provider
• [x] Cloud Pub/Sub Stream Provider #3210

[galvesribeiro]

@gabikliot I didn't found any Stream/Queue service on GCP. The only thing usable for our case is the Cloud DataStore. Do you know anything else?

[galvesribeiro]

Nvm, looks like Pub/Sub is the guy https://cloud.google.com/pubsub/

[gabikliot]

Yes, exactly, pub sub. And datastore for storage.

[galvesribeiro]

We can't release nugets for it since all nugets from Google are on pre-release state :cry:

[gabikliot]

We can have a pre-release Nuget for OrleansGoogleUtils - we already have it this way.
Whether we publish it on Nuget.org (just like datastore is https://www.nuget.org/packages/Google.Apis.Datastore.v1beta2) or not is a secondary question. Even if not, people could still build Orleans themselves and get that Nuget, or take the latest stable CI build.

[galvesribeiro]

Sad thing. The Protobuf nuget that current project in main branch uses, now has a release nuget however, the others we will use in this issue isn't. Anyway, will follow your suggestion once I have everything from AWS done. Thanks

[seniorquico]

@galvesribeiro It sounds like OrleansGoogleUtils will contain both the GCP integration providers and the Protobuf serializer? I have an interest in testing/using only the Protobuf serializer. If these are combined, wouldn't I have to pull all of the other GCP dependencies into my project? If that's the case, I think it would be great if these were split into separate packages. That would also mitigate the issue from your previous comment regarding the Google packages being pre-release except Protobuf.

[sergeybykov]

I concur - looks like having Protobuf integration in a separate binary/NuGet makes a lot of sense, as Protobuf is widely used outside the GCP.

[galvesribeiro]

@seniorquico agreed. With the rename on the projects will better have it on separate projects.

[shlomiw]

Hi,
We're building our back-end with Orleans and we intend to deploy it on GCP, can you please let us know when do you plan adding the support?
Otherwise I probably should use the on-premise setup.
Thanks!

[sergeybykov]

@shlomiw We aren't currently working on adding support for GCP services. We just don't have any internal demand for them. But we'll gladly accept contributions in that direction. What specifically are you looking for? We could help you with writing those providers. It's not terribly difficult.

[shlomiw]

Hi Sergey,

We are considering using Datastore for the Membership/Reminders, and also Pub/Sub for Streaming.
For the grain state we're currently loading it manually at OnActivateAsync and saving when needed.

I saw the AWS utils, yep, as you said it doesn't look difficult. We will consider implementing the GCP utils then we can contribute.

Will let you know, Thanks!

[galvesribeiro]

Please do @shlomiw. Would be nice to have it finished. I just didn't had time to complete GCP and the they don't have trials until you pay for it.

[dVakulen]

@galvesribeiro I recently started free trial on GCP, it lasts for 12 month and/or 300$ spent.

[galvesribeiro]

I guess it is something in Brazil then @dVakulen. They always require credit cards...

[dVakulen]

The credit card is required for avoiding abusive usage. They just locking 1$ for few hour and thats all.

[galvesribeiro]

Yeah but what I said is, that since they don't officially released in Brazil by that time, it wonder accept international cards issued here.

[gabikliot]

@shlomiw, do you plan to run in vanilla VMs on GCE or in app engine?

I would gladly review all your prs and consult on any issue.

Using Google pub sub for streaming should be quite similar to Azure queues. Datastore to membership sounds good too.

[galvesribeiro]

Would be great to have GCE

[shlomiw]

Hi,

Our deployment is on Google VM's. We're hoping for the .Net Core release so we could consider cheaper linux or even Google app engine, which they announced .Net core support (!).

We were thinking of Datastore for the Membership, but now we're considering to just use Cloud SQL, and then we can use the SQL code. What do you think?

For streaming, we will probably implement it with Google pub sub and will galdly contribute.

[gabikliot]

Yes, you can try Cloud SQL. I am not sure if there are any differences in the setup scripts. But even if there are, this should be solvable.

[galvesribeiro]

@shlomiw check #3210

[shlomiw]

@galvesribeiro - yay! perfect! will test it :) thanks - very much appreciated

[galvesribeiro]

@shlomiw the PR is completed. All feedback and tests were addressed. Waiting for review.

Please be aware that #3210 tackle specifically GCP PubSub stream provider which in that particular case is used basically as a queue like in Azure Queue (with a shared subscription in Pull mode). To leverage more extreme scenarios we would need to add GCP CloudIO support which should be added as another stream provider just like Azure EventHub is another as well.

[shlomiw]

@galvesribeiro - your work is extremely appreciated!
In the next few days I'll start reviewing it and later integrate it in our system.
So expect some feedback soon :)
Thanks again! 👍 👍

[gabikliot]

Amazing work @galvesribeiro! As usual.

[galvesribeiro]

PubSub stream provider is merged. Now will find some time for the other remaining providers.

[gabikliot]

Membership with Spanner or datastore would be a natural next choice I think? What do you think?

[galvesribeiro]

I was initially thinking of DataStore. I don't knew about spanner. Will read about it ;)

[shlomiw]

@galvesribeiro - check out https://github.com/NoGame/NG.Data.Spanner - it's our initial work for Spanner EF core provider written by my colleague @tomerpeled .
It's very early since Google haven't yet released an official .Net lib for Spanner. Maybe @gabikliot could push a bit the guys there ;)

[galvesribeiro]

Interesting... In all cases, looks like would be better to start with Grain State, Reminders, Statistics and Membership on top of DataStore. Later on, we could have Spanner provider as well when we have something more stable in terms of client APIs.

[shlomiw]

Ok, great, datastore is also good for the job.
Thanks!

[gabikliot]

It looks like the Spanner .NET repository is very active:
https://github.com/GoogleCloudPlatform/google-cloud-dotnet/commits/master/apis/Google.Cloud.Spanner.Data

I can ask @jskeet about the latest state and how much they are investing in it.
We could do Datastore for now, but longer term I think for GCP Cloud Spanner would be the way to go, due to much richer semantics and global availability and replication support.

[shlomiw]

@gabikliot - thanks for pointing that out! we didn't see the new Spanner .Net update, we're going to check if it's ready for integration. Great news :)

[jskeet]

A couple of points:

• I would strongly advise using the Google.Cloud.Spanner.Data package (ADO.NET provider) rather than using the Google.Cloud.Spanner.V1 package (mostly raw gRPC client). This handles session and transaction creation appropriately, etc. The package is currently in beta.
• As noted, the code is actively being worked on. I'm finding out how open I can be in public about our aspirations.

It's exciting to see the possibilities for uses of Spanner though :)

[tomerpeled]

@galvesribeiro Thanks for your great work!

[tomerpeled]

@jskeet This is great news!
I will integrate it to our EFCore Spanner provider in the near future.

[galvesribeiro]

Ok, if Google.Cloud.Spanner.Data is an ADO.Net provider, I can easily attach it to the relational providers so no need to use a new provider for it.

[jskeet]

@galvesribeiro: It may not be quite that simple. Spanner doesn't currently support DML, so the "CUD" part of "CRUD" has to be handled slightly differently. It can all be done via the ADO.NET provider, but not using "vanilla" SQL. Documentation is still being written around all of this - it's very much under active development. (We're aware the lack of DML causes a bit of an impedance mismatch, but it's the best balance we can strike at the moment.)

[galvesribeiro]

@jskeet ahhh. I see.

In that case, I would follow with the first option just to unblock people like @shlomiw and have the providers to use DataStore for now.

We'll have a second round on GCP providers adding Streaming providers for CloudIO and by that time we can add Spanner for storage and membership. That way you will have time to work as you want on the ADO provider and then we can decide if we will use it or fall back to the lower level APIs with gRPC etc (which is what we did with most of the providers here). What do you think?

[jskeet]

@galvesribeiro: I can definitely see there being room for both Datastore and Spanner providers. I know very little about Orleans to be honest, but in particular if users want to integrate their Orleans storage with other data, it could be useful to have them in the same system...

[galvesribeiro]

@jskeet in Orleans, the storage provider is the component that "talk" to the backend store and persist the Actor State. That state is meant to be accessed from outside Orleans since it would break some concurrency guarantees that Orleans provide with its own threading model.

But yes, we can have both providers, I'm just saying that we could start with DataStore for now, and later on, spend more time on Spanner.

[shlomiw]

@galvesribeiro - I've tested it and it works :) thanks 👍 👍

Few questions if I may:

• Why only Pull mode? what is the pulling intervals?
• How do I support ack/nack in a way that I want to process an item, and take it out from the queue only if the processing was successful? (one of the most important reasons why we need durable queues)

Thanks again!

[gabikliot]

@shlomiw, ack is automatically done by Orleans. If the grain processed the steam item without throwing an exception, it will be acked and deleted from the queue. Otherwise it will be retried.

As for push/pull, Orleans streaming only supports pull so far.
Interval: it is configurable. Once it finds a msg in queue it pulls them all immideately until queue is empty. But if the queue is empty, it retries after an interval.

[galvesribeiro]

Hehehe, @gabikliot was faster than myself.

More one reason for push/pull, is that push (in this particular case) would require an HTTPS endpoint to be open on each silo and there can be multiple Orleans stream agents (the internal SystemTarget that consumes messages from the queue) and it will eventually hit TCP port collision. Another reason is that Orleans deal with backpressure which would be complicated with push.

[shlomiw]

ack is automatically done by Orleans. If the grain processed the steam item without throwing an exception, it will be acked and deleted from the queue. Otherwise it will be retried.

@gabikliot - this is what I hoped for :) raise an exception for nack - great! I'll try to find in the docs/source-code the retries policy.

About the pull/push - I understand. @galvesribeiro - thanks for the additional information.
I'll see how it meets my requirements. For gaming messages - no doubt I have to use the SMS provider (I'm using streams instead of observers to send messages from the game grains to the front-end then by web-sockets to the players).
I'm going to use PubSub only for the durable operations I need - i.e. saving game-events, or important 'transactional' messages between the grains, which I want it to be more robust in case I can't send a message to a grain.
I also intend to use PubSub later to send data to BigQuery :) hopefully when time will come ;)

[galvesribeiro]

@gabikliot / @shlomiw / @jskeet

I'm writing the Storage/Membership/Reminder providers for Datastore and I hit a wall here which is probably my lack of knowledge on GCP tools and you guys may shed some light here...

As part of those providers, I need to have a way to perform conditional writes to ensure consistency but I can't see whether Datastore natively support it like AWS DynamoDB or Azure Tables. Ofc I can read before write and compare the ETag property in a transaction scope, but that is not the ideal.

In this particular doc, it state:

When a transaction starts, Cloud Datastore uses optimistic concurrency control by checking the last update time for the entity groups used in the transaction...

However, looks like the etag/timestamp field is not exposed in the Entity object so I can't return it back to Orleans.

I appreciate any guidance.

Thanks!

[gabikliot]

I am not an expert on Datastore (I know more about Spanner), but it looks like you will need to use transactions and set them correctly for it to work with Orleans table API:

1) add etag number to your data explicitly (just a counter)
2) set mode to TRANSACTONAL:
https://cloud.google.com/datastore/docs/reference/rpc/google.datastore.v1#google.datastore.v1.CommitRequest.Mode
3) in the tx read it and compare to what Orleans passes and increment if equal or Abort the tx if not and return the stored value (let Orleans retry on etag mismatch)
4) in the result:
https://cloud.google.com/datastore/docs/reference/rpc/google.datastore.v1#google.datastore.v1.MutationResult
if you get conflict_detected, don't retry. Just return false.

I think that is what we did in ZK. It also did not have eTags, but we simulated it.

Basically, you want to simulate/emulate Orlean's table semantics with eTag and let Orleans retry.

[gabikliot]

Now I remember. It was not in ZK, but in SQL provider. We had to imitate the etag in SQL provider. So just take a look there and follow its pattern.

https://github.com/dotnet/orleans/blob/master/src/OrleansSQLUtils/CreateOrleansTables_MySql.sql#L538
Notice how Version is compared and how it is manually auto-incremented in the tx code.

[ghost]

hi @galvesribeiro
Have you made progress with storage/membership/reminder providers or do you have any existing code that could be used as a basis? Or have you come to the conclusion that it is not suitable?

[galvesribeiro]

@lawgorht sorry for the delay. I've put this issue in freeze as all efforts were towards Orleans 2.0.0 release. Now that it is gone, we should get back to it.

I'm in out-of-office until next week. As soon as I get back will check on this and report back.

Thanks

[ghost]

sounds good, thanks for all your efforts. I asked because we might have to use gcp for a project and it would be good to have suitable persistence layers.

[sergeybykov]

We decided to track with issues individual providers that somebody is working on. Hence, closing the grouping issue. Please open issues for specific providers you are interested in contributing.