Efcore: Cosmos DB provider

[grahamehorner]

Note: This issue is for discussion and questions about the EF Core Cosmos provider.

Issue #12086 is tracking the work being done on the implementation.

---

Please consider creating entity framework core providers for Cosmos DB.

[divega]

We should consider investing on this. Keeping it in the backlog for now because we don't have short term plans for it.

[divega]

We are going to start investigating this in 2.1. We have a lot of work to do to figure out what the goals should be for EF Core support for Cosmos DB in the short and long term, e.g. from answering basic questions like what value EF Core support should bring to the table to which exact APIs of Cosmos DB we should support.

[MisinformedDNA]

This issue encompasses #1142, so I think #1142 can be marked as a duplicate.

[grahamehorner]

I would love to see support doe take, skip foe paging as at present Cosmos doesn't server side support paging as I understand it

[divega]

@MisinformedDNA from our initial discussions with Cosmos DB, we are not likely going to target the Table Storage API. We could close #1142 because we are not doing it, but it is not necessarily encompassed.

[DavidBurela]

Now that EF core 2.0 has been released. Hopefully this can get considered for 2.1 now <3
Been looking forward to it since 2014 when the original #1035 was created

[MisinformedDNA]

@divega Cosmos DB is essentially four different APIs to do different things. I would expect either a single Cosmos DB provider to handle all four APIs (seemingly this issue) or four separate Cosmos DB providers, one for each of Cosmos DB's APIs. The latter seems more reusable, composable and maintainable.

[satyajit-behera]

This will be an interesting feature. Just curious, any planned date for 2.1 release of EF? Will be helpful for new projects with more interesting use cases.

[ErikEJ]

@satyajit-behera The EF Core roadmap follows the ASP.NET Core roadmap, which you can see here: https://github.com/aspnet/Home/wiki/Roadmap (or maybe the .NET Core roadmap??? https://github.com/dotnet/core/blob/master/roadmap.md )

[VagueGit]

Neither of those roadmaps acknowledges the existence of Cosmos DB. Does this mean there is no plan for EF to support Cosmos DB?

[smitpatel]

@VagueGit - This is roadmap of EFCore https://github.com/aspnet/EntityFrameworkCore/wiki/roadmap which outlines which features EF Core team is going to implement. Release cycle of EF Core is part of large org with ASP.NET & .NET. Timeline for release (not the features included in EF Core for release) would be found in their roadmap (in the links posted by @ErikEJ )

[rosieks]

What about migrations? Are you going to support it too?

[smitpatel]

@rosieks - Since CosmosDb stores schema-less JSON documents, there are not schema changes in code model which needs to be scaffolded in a migration and applied to the database. You can continue using the same database to store changed model.
What are the scenarios where you believe we would need migrations support? We would be happy to analyze and see what kind of support would be good for such scenario (migrations or may be something else)

[rosieks]

I agree that CosmosDb is schema less but it doesn't mean there is no schema at all - we still have model in the code that changes. I thought about support for changes in model. For example when I rename class or property I would like to update existing documents in database. The same if I change type or introduce new field - I would like to provide some logic to convert/fill it.

[smitpatel]

@rosieks - That is useful scenario though it is data migration (rather than schema migration). At present the support for data migration is very less in EF Core in general. So it would be kind of a new feature altogether. I believe, such data migration/manipulation can be done in cosmos db using stored procedures. Priority of it happening in intial preview is very low.

[MisinformedDNA]

We should create a new issue for supporting data migration. Personally, I think it is a valuable feature, but it isn't essential to the initial provider.

[JohnGalt1717]

Data Migration is vital for Cosmos. Here's why:

Because it's a json store and schema-less (sort of) you tend to denormalize your data. I.e. you'll duplicate a lot of data to eliminate the need for joins across tables.

What happens often is that you'll discover that you'll need a new field on existing data and need to add that field to every record. So migrations should allow for the addition of the new field AND the population of that field automatically. Typically this would be a series of queries that gets the values and then iterates over the records and updates the values and then saves them back.

Please keep this in mind while developing the migration scenarios as this is a standard process that must happen all of the time.

[smitpatel]

@JohnGalt1717 @rosieks Filed #11099 to track data migration related work for Cosmos Db.

[benmccallum]

Loving the potential for an EF provider here. Currently investigating GraphQL + CosmosDB and I think it'd be nice to use EF's programming paradigm rather than Cosmos DB's SQL or Linq providers. Struggling a bit getting graphql-net playing nicely with relations and show forth. Swapping in an EF provider rather than the Linq provider could be my answer as it's intrinsically know the relationships/navigation properties to follow and translate.

Question, how will nesting be handled/annotated?

For instance, take a blog data model... If you wanted to take advantage of document db and store a Post's Comments in the same document (nested under a property called "Comments" rather than off in another collection linked by "foreign keys"), how would you achieve that?

Or... if you just had a User with properties like Profile and Settings which were themselves POCOs that logically grouped other properties (Profile.Name, Profile.Age, Settings.Notifications.ReceiveEmails, Settings.ShowRealName) and should be stored in the same document in Cosmos.

Right now, the way EF works these would automatically become navigation properties, right?

[anpete]

@benmccallum, @natemcmaster and I hacked something up ages ago: https://github.com/anpete/GrefQL

[divega]

how will nesting be handled?

@benmccallum the idea is to use ownership to model this. When working with relational databases owned entities works as a relationship with stronger semantics (owned entities cannot exist on their own, and we set up eager loading by convention on them). With document databases, ownership will be used to signal that the relationship is mapped within the same document/aggregate rather than as an FK.

Currently we support owned references (similar to EF6 complex types), and we are missing owned collections (tracked as #8172).

[benmccallum]

Thanks @divega, great explanation and sounds like a great solution. Was poking around the source code thus far and looks awesome. 😄

Looks like Cosmos DB doesn't support cross-collection joins, there's talk of cross-document joins which might help, but I guess this means any navigational properties (FKs) would always be lazy-loaded? Could this be an n+1 nightmare? I'm still getting my head around doc dbs but it seems like putting commonly queried data into a single doc, even if that involves duplication, is the way to go. I'm just struggling with scenarios where nested collections could involve thousands of items as this apparently hits document size limits and drastically affects perf. I guess you need to move them out to their own collection and live with an n+1 scenario, ensuring you have aggregate data like counts, etc. being tallied on the parent doc.

@anpete, nice, will take a look now, thanks!

[JohnGalt1717]

Is there any alphas or early builds that we can pull of this on myget or something?

[ajcvickers]

@JohnGalt1717 Not yet, but it is a priority for us to get to that point soon.

[JohnGalt1717]

@ajcvickers Thanks, good news. I'm starting a new project that is huge and will be long running (and will be a while before it's in production) and I'd rather be using the Entity Framework approach than DocumentDbClient. I also hope that you'll have things like FirstOrDefaultAsync etc. working because the DocumentDbClient is BRUTALLY basic with what it supports.

[dilmerv]

Do you guys have a ETA on when the "Cosmos DB provider" will be available? Thanks!

[JohnGalt1717]

And is it possible to compile a working-ish cosmosdb provider yet? Dying here!

[vertonghenb]

Any intention to support the Graph setup via Gremlin queries?

[JonPSmith]

A question. Will you be building a default ExecutionStrategy for the cosmos provider in a similar way to SQL Server's SqlServerRetryingExecutionStrategy or do we need to write that ourselves?

[ErikEJ]

@JonPSmith I think it is already built into the Cosmos SDK - but of course it would be nice to be able to configure the options https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.connectionpolicy.retryoptions?view=azure-dotnet

[JonPSmith]

Thanks @ErikEJ,

I had a look at the actual code and there is a CosmosExecutionStrategy. so the answer to my question, yes there is a ExecutionStrategy for the cosmos provider but there isn't an extension like SQL Server has. Here is how you add the strategy.

var builder = new DbContextOptionsBuilder<NoSqlDbContext>()
    .UseCosmos(
        "urltocosmos",
        "somelongkey",
        "databaseName",
        options => options.ExecutionStrategy(c => new CosmosExecutionStrategy(c)));

[AndriySvyryd]

@JonPSmith Cosmos will use a retrying execution strategy by default, so the user doesn't need to specify one unless they need to change the way it behaves.

[JonPSmith]

@AndriySvyryd. Thanks for letting me know about that as I am writing an article on EF Core and Cosmos DB and was going to put the ExecutionStrategy on my example

PS. It makes sense that retries on with Cosmos as it doesn't support transactions, so there is no down side by having retries on.

[Mortana89]

@AndriySvyryd What about retries for 412? Will we support ETag concurrency in the cosmos DB provider?
I'm trying to see how I can implement it myself, but I'm a bit pestered with the fact that you could have partial saved state in the db (if, e.g. 3 changes are sent to the DB, but the 3th got a 412 due to different ETag).
Without worrying about that case, would it be a good idea to throw a dbconcurrency exception from the cosmosclient, when we received a 412? And then catch this in the callers code by using something like Polly? I would then only need to add a shadow prop for the ETag value, but it would be a begin to implement concurrency mechanisms. What are your thoughts?

[AndriySvyryd]

@Mortana89 We'll likely throw a DbConcurrencyException, but it would be up to the caller to decide how it should be handled - like resending updates for all entity types overriding any changes or querying the updated data, determining which ETags are mismatched and merging the changes.

[Bassist067]

Hi everyone! I'm using the CosmosDB EF Core Provider 2.2.4 and here's the strange issue - EF Core seems to ignore the [JsonProperty(NullValueHandling = NullValueHandling.Ignore)] attribute when I try to write null in Cosmos.

So the next time I try to deserialize it, I've got Unable to cast object of type 'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JObject'. It seems like when I deserialize the null value to some class model, it's crashing.

So the questions -

• how can I force EF Core to ignore nulls when I write into Cosmos?

• how can I deserialize object with one of the properties set to null?

[AndriySvyryd]

@Bassist067 The Cosmos provider currently doesn't handle any attributes. Properties of nullable type generally accept nulls by default.
You are using an early preview version of the provider, please try 3.0.0-preview5 and if you are still running into this open a new issue.

[AndriySvyryd]

All the feature work planned for 3.0 has been completed.
Open new issues for any further questions.

[githubavi]

Hi, Any plan to support EF Core for Cosmos Db Gremlin Api?

[ajcvickers]

@githubavi Currently no plans.

[richardrobberse]

EDIT: apparently the ETag concurrency token was implemented recently by this PR: https://github.com/dotnet/efcore/pull/19581
@jviau @AndriySvyryd can you guys shine some light on how to effectively use the ETag concurrency token? Thanks in advance!

Few questions that popped-up after creating a simple CRUD REST API with CosmosDB:

• How to implement concurrency handling with ETags?
  Does the provider take the ETag on the model into account during updates or should I build the ETag comparison myself?

• I've added the _etag to my model and it is returned when querying for entities, so far so good.
  When I, for example, add a new record through the API the _etag property is not set on the object after calling Add() and SaveChanges(). I kind of expect this to work the same way as it works with the Id property.
  Is there a way to get the ETag when adding a new record without retrieving the same record again?

[JohnGalt1717]

@richardrobberse CosmosDB provider isn't even remotely close for being used in a realworld use case. Maybe .net 5.

[richardrobberse]

@richardrobberse CosmosDB provider isn't even remotely close for being used in a realworld use case. Maybe .net 5.

That's quite a statement

[JohnGalt1717]

@richardrobberse Look at all of the basics it doesn't support (try sub arrays of objects the CosmosDb way, I dare you) and tell me that I'm wrong. You can't take any marginally complex CosmosDb database with multiple collections and make it work in EF Core 3.1 as it stands today. The basics (like concurrency) aren't there.

[richardrobberse]

@JohnGalt1717 I'm not in need of anything complex, I just need a simple CRUD API so I think I'll manage once the concurrency gets added (hopefully in 3.1.2)
If you have other suggestions, please let me know!

Have you tried to use the CosmosDB SDK? There's not much documentation on it but I managed to get it to work at some point, I wasn't satisfied with my implementation so I kind of moved away from it.

[JohnGalt1717]

@richardrobberse Just use direct CosmosDb table api with linq until they get it working right and fully support child lists. (i.e. right now you can't even do Contact.Addresses properly with it, so your basic CRUD will probably fail too)

[jviau]

EDIT: apparently the ETag concurrency token was implemented recently by this PR: #19581 @jviau @AndriySvyryd can you guys shine some light on how to effectively use the ETag concurrency token? Thanks in advance!

Few questions that popped-up after creating a simple CRUD REST API with CosmosDB:

• How to implement concurrency handling with ETags?
  Does the provider take the ETag on the model into account during updates or should I build the ETag comparison myself?
• I've added the _etag to my model and it is returned when querying for entities, so far so good.
  When I, for example, add a new record through the API the _etag property is not set on the object after calling Add() and SaveChanges(). I kind of expect this to work the same way as it works with the Id property.
  Is there a way to get the ETag when adding a new record without retrieving the same record again?

When it is released, Cosmos ETag concurrency will be handled automatically when a property, shadow or concrete, is mapped to the JSON property _etag and is marked as a concurrency token. Anything mapped to "_etag" must be either string, or have a string value converter. There are helpers for this:

entityBuilder.UseEtagConcurrency(); // adds a shadow property for "_etag" as a concurrency token
entityBuilder.Property(x => x.MyProperty).SetEtagConcurrency(); // maps to "_etag" json property, sets concurrency token

There is no convention for automatically detecting properties named "etag" or anything similar and making it the concurrency token, it needs to be set explicitly in the fluent API for now.

When an entity has an etag concurrency token, it will automatically be included in the ItemRequestOptions.IfMatch for all applicable requests to cosmos (insert, replace, delete), so you will start seeing DbUpdateConcurrencyException on conflicts during SaveChanges.