Runtime: Create System.Spatial types for SQL-Independent geography code

Created on 24 Sep 2016  ·  51Comments  ·  Source: dotnet/runtime

It's been about 8 years since Spatial support was added to SQL Server. At the time, .NET was not as far along as it is today, and was not open source. The team created a SQL-dependent version of the Geography classes, which has created situations this this at hundreds of companies.

Now that we have .NET Core, Roslyn, and a bunch of other new, amazing technologies in .NET... it might be time to give it another go and see if we can create a vanilla set of System.Spatial types that the Entity Framework, OData, and other spatial libraries can inherit from.

Since DbGeography types aren't even really on the EFCore schedule yet, this is the perfect time to get something spun up so that EFCore can leverage these types when the time comes.

api-needs-work area-Meta

Most helpful comment

Fantastic questions, this is a great start. I have some thoughts, but I realize they are just a personal perspective. I'll see if I can explain a scenario that I find to be pretty common, mostly because I maintain several apps with a similar architecture.

The Problem

I have an app that runs a SQL Database, and its persistence layer is powered by Entity Framework 6. I personally am a "schema-first" developer, meaning I:

  • build the database tables for what I need
  • I have an EDMX file pointed to the database that gets automatically updated.
  • I have T4 templates that generate different parts of the code architecture from the EDMX file.

As a part of that code generation process, I always have a project called ProjectName.Core, which holds the code-generated objects + any additional objects that need to be shared by multiple app assemblies. This library MUST HAVE few dependencies as possible, because it could be used in a Xamarin app, or a Website, or a UWP app.

From there, I use a straightforward "Database-as-a-Service" architecture, powered by Restier and OData. Within 15 minutes of designing a database, I can be querying it against a REST service and have it wired into .NET-powered front-end.

Ok, so that's the mental baseline for the architecture. Restier => OData => "Core" Object Library => Entity Framework => SQL Server.

NOW, let's say the app I'm building gathers the exact geographic coordinates for every apartment complex in the US. So I'd have an "Address" table in the database, and I'd need to add a "Geography" column to that table to store either the pinpoint location, or an outline of the property boundaries. EF6 has a built-in type for this called System.Data.Spatial.DbGeography.

But in order to map this generic spatial type to SQL Server, it also needs access to the Native SQL Server types. This native DLL is conveniently packaged in the Microsoft.SqlServer.Types package on NuGet, and has to be added to any project that does Data Access on Azure Web Apps.

So we've added the column. Now we must update the EDMX model, run the T4 templates, and compile. But now, after doing so, the ProjectName.Core package will not compile, because DbGeography is in the EntityFramework assembly. Which means the Core project is now _dependent on the Entity Framework_ to be used by any other project, including a WPF or UWP project.

This, in and of itself, is a HUGE no-no. It's reminiscent of the Windows days before MinWin (or .NET Core) when types and architectural layers were a spaghetti mess.

To make matters worse, the OData Team (which used to be a bigger group but is now "in-sourced" to a super-small team in China responsible for WebApi, OData, and Restier) has their OWN way to map Spatial data types. It's the Microsoft.Spatial library, formerly known as System.Spatial). It's a great library. But AFAIK, it's pretty much ONLY used by OData.

Summary so far: If you're using WebAPI OData + EF + SQL Server for your geographic queries, you are dependent on a _minimum_ of 3 different DLLs that essentially do the same thing.

Impact

If you're still with me, let's go one step further: Let's say you want to upgrade your "Core" object assembly to .NET Standard. You can't. Because EF Core does not have DbGeography support yet. And you can't use Microsoft.Spatial (or any of the other libraries mentioned), because they doesn't map directly to SQL Server.

Your only option (at least, the only one I could find) is to remove the DbGeography property from your core model, and use a database trigger to parse the "WellKnownText" (a standards-based text representation of the geographic data) to your "Geography" column to be able to run queries like "show me all apartments within 50 miles of Phoenix, AZ."

I know this was long, but hopefully that provides an explanation of the depth of the problem.

The Way Forward

At this point in time, we're at a crossroads. The next version of SQL Server is underway, EF Core "Geography" support will begin work soon, and .NET Standard 2.x is in progress.

SO what I am proposing is this:

  • Get the SQL Server, Entity Framework, and OData teams to agree _in principle_ that 3 DLLs to accomplish the same task is not the right answer for their customers.
  • Promote the artist formerly known as Prince library formerly known as System.Spatial to ".NET Standard" status. This means it's considered a first-class part of the .NET Framework, and primary responsibility for maintaining it moves away from a team of junior devs in China.
  • Upgrade the System.Spatial to code to compile with .NET Standard.
  • Have the EF Core team use System.Spatial.Geography instead of System.Data.Spatial.DbGeography for their new Spatial implementation.
  • Get the SQL Server team to replace their Microsoft.SqlServer.Types implementation with the new System.Spatial, since .NET can run inside the SQL Server process (and a .NET Standard-compliant binary should be able to run inside the Linux version of SQL Server too).
  • Get the other Spatial library creators to help make sure System.Spatial has the design they need to be able to inherit from and add value to the platform.

From a marketing standpoint, all of this can be spun as a needed upgrade that helps enable the next generation of Spatial apps that will leverage things like location tracking, AR, and IoT to create great customer experiences.

LOL when I started this I didn't think it was going to end up being a blog post. But I hope it helps understand where we are at, and where we could be going.

All 51 comments

Why not just use the GeoApi library as a base and build on it.

@robertmclaws is GeoApi something that could be leveraged forwarded to instead of building yet another solution in .NET Core (disclaimer: I have no background in the geo space, just fishing for information)

For completeness, Microsoft developed another set of spatial types known as System.Spatial / Microsoft.Spatial. The code lives here: https://github.com/OData/odata.net/tree/ODataV4-7.x/src/Microsoft.Spatial (thanks @robertmclaws for the link).

@divega did you mean Microsoft.Spatial?

Given that there are already solutions, I don't think it makes sense to add more such types into corefx. Closing.

If you think I am wrong, please say so - please also explain what do you expect in corefx in such case.

Please do not close this issue. The point was not to create/identify a "new" solution. The point was to promote a solution for "Spatial" to full-fledged ".NET Framework" status, meaning that more code besides OData should be relying on it. Right now, that namespace is "Microsoft.Spatial". It needs to be promoted to "System.Spatial" and needs to be used as the core library for all Spatial types in the code Microsoft ships. Including SQL Server. That way, base object libraries can take a dependency on THAT library, and not take dependencies on provider-specific implementations (like DbGeography). This is CRUCIAL for cross-platform development.

@karelz It was called System.Spatial at some point :smile:

I agree with @robertmclaws this issue deserves more discussion.

Spatial seems to me to like a specialized type. At minimum we need people/teams who use it heavily to design it -- design it in such a way it can be reused as suggested above. @divega is it something you could drive?
It should probably also sit on top of corefx libraries, so that it can be reused across .NET platforms (like ImmutableCollections). It can easily live in any repo, incl. corefx - I don't have strong opinion on that yet.

cc: @terrajobst @danmosemsft @weshaggard for opinion here ...

@karelz Why do you say it's a "specialized" type? To me it is no more specialized than System.Threading.Thread. Every type has a reason for its existence.

@robertmclaws I treat it as specialized type, because I don't believe it is widely used by applications/frameworks.
As such it would be IMO better to build it in a separate NuGet package. Potentially not even from corefx repo.

@karelz DbGeography is a pretty widely used EF type. What I'm suggesting is to take the existing System.Spatial library, put it in a different repository (possibly .NETFX, possibly in its own separate repo), make it .NET Core compliant, and mandate it as the foundation for all location-based code Microsoft ships. It would save a lot of architectural hassles that you, not being familiar with the issue, are probably not aware of (and I mean no disrespect with that statement).

HTH!

I treat it as specialized type, because I don't believe it is widely used by applications/frameworks.

That's a very narrow view. The perceived lack of widespread use in .Net could be because developers needed to develop almost everything from scratch in this domain.

A lot of the web and mobile app development relies on GPS location and map display, currently vendors prefer Java & Python because of the community support available and libraries that are taken as defacto standards. GeoApi, NetTopologySuite, Proj4Net, DotSpatial etc are examples where we have taken from existing Java libraries, these as well as the ORM's like EntityFramework, NHibernate etc. will greatly benefit if base spatial types are fixed.

Adherence to the OGC specs is a must have minimum.

As such it would be IMO better to build it in a separate NuGet package. Potentially not even from corefx repo.

+1 for that

@robertmclaws, thanks for explanation, I am trying to understand here what needs to happen where - for which I need to understand a bit the space, current limitations and goals you're trying to achieve. So let me ask you for more details:

What I'm suggesting is to take the existing System.Spatial library, put it in a different repository (possibly .NETFX, possibly in its own separate repo)

Why do we need to move it out? Is too tied to OData? Would it be possible to leave it in OData repo and just produce yet another NuGet package for just System.Spatial from there (assuming that's not the case today - otherwise we wouldn't have this discussion I assume).

make it .NET Core compliant

By that you mean, make it .NET Standard compliant, right? So that it can be used anywhere - full .NET Framework, Xamarin, .NET Core, .NET Native.

mandate it as the foundation for all location-based code Microsoft ships

For that I think System.Saptial has to satisfy all the needs of current one-off implementations used around MS (it is not clear to me if EF has one or not today). We will need someone familiar with each of them (or the most important ones) to assess if System.Spatial is the right thing and then figure out transition plan -- e.g. I assume we cannot just nuke the existing ones if there are libraries and apps in the ecosystem using them.
@divega would you know who are the right parties to bring to the table?

It would save a lot of architectural hassles that you, not being familiar with the issue, are probably not aware of (and I mean no disrespect with that statement).

I am pretty sure that other folks on CoreFX team are not experts in the area either, so the key thing is to find the right folks who can fully assess usefulness of this approach and put it in motion, otherwise this issue will just sit here ignored, because we won't know what to do about it.
I am really hoping that @divega can help here :) ... if not, I will need more insights into today's MS technologies in the Spatial space to attempt to find the right people ...
BTW: No offense taken. Just please be patient with my lack of knowledge.

@suryapratap

The perceived lack of widespread use in .Net could be because developers needed to develop almost everything from scratch in this domain.

I wonder why none of the 4-ish attempts of the APIs are reusable elsewhere. Are they specialized? Are they incomplete? Did they grow in parallel at the same time and therefore we have fragmented .NET story?
If I need Spatial types in my app, why cannot I just pick one of the 4 libraries you listed and use types from there? Does anything prevent me to do that?
Or is the biggest problem the fragmentation?

Adherence to the OGC specs is a must have minimum.

No idea what that is, but I assume all the solutions listed above are compliant with the spec already, right?

@karelz I am happy to drive this. Although I am no expert myself on the subject, I might have enough context to help and I am gathering some data. I actually think your questions are a great start.

Fantastic questions, this is a great start. I have some thoughts, but I realize they are just a personal perspective. I'll see if I can explain a scenario that I find to be pretty common, mostly because I maintain several apps with a similar architecture.

The Problem

I have an app that runs a SQL Database, and its persistence layer is powered by Entity Framework 6. I personally am a "schema-first" developer, meaning I:

  • build the database tables for what I need
  • I have an EDMX file pointed to the database that gets automatically updated.
  • I have T4 templates that generate different parts of the code architecture from the EDMX file.

As a part of that code generation process, I always have a project called ProjectName.Core, which holds the code-generated objects + any additional objects that need to be shared by multiple app assemblies. This library MUST HAVE few dependencies as possible, because it could be used in a Xamarin app, or a Website, or a UWP app.

From there, I use a straightforward "Database-as-a-Service" architecture, powered by Restier and OData. Within 15 minutes of designing a database, I can be querying it against a REST service and have it wired into .NET-powered front-end.

Ok, so that's the mental baseline for the architecture. Restier => OData => "Core" Object Library => Entity Framework => SQL Server.

NOW, let's say the app I'm building gathers the exact geographic coordinates for every apartment complex in the US. So I'd have an "Address" table in the database, and I'd need to add a "Geography" column to that table to store either the pinpoint location, or an outline of the property boundaries. EF6 has a built-in type for this called System.Data.Spatial.DbGeography.

But in order to map this generic spatial type to SQL Server, it also needs access to the Native SQL Server types. This native DLL is conveniently packaged in the Microsoft.SqlServer.Types package on NuGet, and has to be added to any project that does Data Access on Azure Web Apps.

So we've added the column. Now we must update the EDMX model, run the T4 templates, and compile. But now, after doing so, the ProjectName.Core package will not compile, because DbGeography is in the EntityFramework assembly. Which means the Core project is now _dependent on the Entity Framework_ to be used by any other project, including a WPF or UWP project.

This, in and of itself, is a HUGE no-no. It's reminiscent of the Windows days before MinWin (or .NET Core) when types and architectural layers were a spaghetti mess.

To make matters worse, the OData Team (which used to be a bigger group but is now "in-sourced" to a super-small team in China responsible for WebApi, OData, and Restier) has their OWN way to map Spatial data types. It's the Microsoft.Spatial library, formerly known as System.Spatial). It's a great library. But AFAIK, it's pretty much ONLY used by OData.

Summary so far: If you're using WebAPI OData + EF + SQL Server for your geographic queries, you are dependent on a _minimum_ of 3 different DLLs that essentially do the same thing.

Impact

If you're still with me, let's go one step further: Let's say you want to upgrade your "Core" object assembly to .NET Standard. You can't. Because EF Core does not have DbGeography support yet. And you can't use Microsoft.Spatial (or any of the other libraries mentioned), because they doesn't map directly to SQL Server.

Your only option (at least, the only one I could find) is to remove the DbGeography property from your core model, and use a database trigger to parse the "WellKnownText" (a standards-based text representation of the geographic data) to your "Geography" column to be able to run queries like "show me all apartments within 50 miles of Phoenix, AZ."

I know this was long, but hopefully that provides an explanation of the depth of the problem.

The Way Forward

At this point in time, we're at a crossroads. The next version of SQL Server is underway, EF Core "Geography" support will begin work soon, and .NET Standard 2.x is in progress.

SO what I am proposing is this:

  • Get the SQL Server, Entity Framework, and OData teams to agree _in principle_ that 3 DLLs to accomplish the same task is not the right answer for their customers.
  • Promote the artist formerly known as Prince library formerly known as System.Spatial to ".NET Standard" status. This means it's considered a first-class part of the .NET Framework, and primary responsibility for maintaining it moves away from a team of junior devs in China.
  • Upgrade the System.Spatial to code to compile with .NET Standard.
  • Have the EF Core team use System.Spatial.Geography instead of System.Data.Spatial.DbGeography for their new Spatial implementation.
  • Get the SQL Server team to replace their Microsoft.SqlServer.Types implementation with the new System.Spatial, since .NET can run inside the SQL Server process (and a .NET Standard-compliant binary should be able to run inside the Linux version of SQL Server too).
  • Get the other Spatial library creators to help make sure System.Spatial has the design they need to be able to inherit from and add value to the platform.

From a marketing standpoint, all of this can be spun as a needed upgrade that helps enable the next generation of Spatial apps that will leverage things like location tracking, AR, and IoT to create great customer experiences.

LOL when I started this I didn't think it was going to end up being a blog post. But I hope it helps understand where we are at, and where we could be going.

I wonder why none of the 4-ish attempts of the APIs are reusable elsewhere. Are they specialized? Are they incomplete? Did they grow in parallel at the same time and therefore we have fragmented .NET story?

@karelz the 4 attempts are not divergent. GeoAPI.NET is an attempt implement OGC GeoApi specs (reference implementation done in Java) it forms the base for the other projects which are basically ports of the JTS (Java Topology Suite). Proj4Net is a spatial projections library, so is DotSpatial.Projections. Spatial4N is a port of Spatial4J.

The point is all of these take up GeoApi as kind of base on which the implementations are done so as to get maximum conformance with the OGC Specs.

Adherence to the OGC specs is a must have minimum.

No idea what that is, but I assume all the solutions listed above are compliant with the spec already, right?

It would do a lot of good if spatial libraries are built conformant to OGC specifications.

My guess is that an open source implementation of .Net spatial library should be database agnostic, hence kind of concerned about use of Microsoft.Spatial.Types as a base. Ideally it would have been better if a common base is provided for all providers and the DBGeography/DBGeometry become specialized implementations. EntityFramework too has this same kind of requirement since an ORM should be database agnostic.

Microsoft.Spatial is a good candidate (My bad I did not see this library earlier). I did not see a support in API for coordinate system transformations though, this could be because OData would be concerned only with transfer of data and not analysis.

so, it's been a while. Has anyone begun looking into this beyond this conversation?

@robertmclaws wow - totally awesome explanation _and_ suggestions.

That's _exactly_ what needs to be done IMO. I've being doing sql spatial since it was added in sql 2008 beta (urgh so long ago, i can't remember) so I've been through hell and back with this :(

If i can +1-bazillion your post, I'd do just that.

It's so frustrating how this (i.e. Spatial in the MS suite) feels like it's always a can't-be-bothered-with feeling and always some random added on thing which has led us to this frustrating place, today.

Please give this some much needed love and resourcing. It's painful out here in the spatial-battle front lines of war :(

Any MS-info-update to help soothe some wounds ....

The point of special types @karelz is ability to perform operations. Distance between A and B to start with but probably more importantly it would be cool if you can for example pick points within given radius with LINQ.
This can still possibly live outside of core as separate NuGet package.
Use of geographical data is getting increasingly common.

How about adding spatial types to .NET standard, but not shipping an implementation of all the methods. Just ship with the ability to read and write text and binary representations of the spatial data. Perhaps all the .NET Standard types are Abstract, or otherwise require registering some 3rd party implementation.

At runtime you could plug-in Microsoft.SqlServer.Types, or something else if you actually need to perform spatial operations. But if you don't you get trivial implementations that throw NotImplementedException when calling any method not concerned with reading or writing a WKT, WKB or GML.

This enables CRUD operations, and it enables EF to translate LINQ queries to SQL, without owning a whole spatial library.

Well, that could be fine, you could have an Abstractions library just like .NET Core's DI implementation, and that could be the lightweight stuff.

I think we're both saying the same thing... my point though is that all of that we don't need a complicated architecture to make this work. EF should handle mapping Spatial types to SQL -> I should never see DbGeography in ANY of my code. And I should be able to read and write WKT, WKB, and GML without needing a database.

Right and EF just needs types to enable developers to write queries over spatial data, and the ability to read and write to the database. It does not need an implementation of STBuffer(d) or STConvexHull().

Personally, I would want the Union and Intersection operations in a library that's more portable than Microsoft.SqlServer.Types (i.e. more mobile/xamarin-friendly).

My team currently has some homegrown methods for doing those operations with multipolygons on a mobile app, and the algorithm isn't the fastest.

Personally, I would want the Union and Intersection operations in a library that's more portable than Microsoft.SqlServer.Types (i.e. more mobile/xamarin-friendly).

My team currently has some homegrown methods for doing those operations with multipolygons on a mobile app, and the algorithm isn't the fastest.

@maf1024: see NetTopologySuite. The latest prerelease packages have targets for .NET Standard 1.0 and 1.3.

Just like @suryapratap said, please don't forget coordinate system projections/transformations, there are many different spatial references identified by SRIDs that one would want to convert between. Also when talking about anything spatial please look to the giants out there in this problem space and try to learn from them (Including their mistakes). Like ESRI ArgGIS .NET SDK, GE Smallworld, and a very amazing but less known SAFE FME. I know it's likely a pipe dream but I would love to see first class support for reading Shapefiles and as many others pointed out, some of the many Open Geospatial Consortium (OGC) specs are great like Simple Features for SQL Specification extended by SQL/MM ISO Spec, GML and KML. It also seems that any library that does not support GeoJSON/TopoJSON would be very much incomplete. GeoRSS is also important.

I would also like to point out that 3-dimensional spatial data is becoming (already) very important, especially in the industry I work in, Smart Cities and Smart Grid around Skyscrapers and Smart Buildings. Really 4 dimensions given all the sensor measurements have a time element as well. 2D was okay for the technology of the past but with Hololens, Windows Mixed Reality, and all the User Experience advancements, visualizing time oriented 3D spatial data is a real necessity.

I tried building Geometry and Geography and the builders from the Microsoft.SqlServer.Types assembly as a .NET Standard library here: https://github.com/dotMorten/Microsoft.SqlServer.Types
It does most of the creating and reading the types: (note: No support for the spatial operations and curve support is only half-way there). It seems to work with most of the data I tried throwing at it.

See also opengeospatial/geoapi#35

I tried building Geometry and Geography and the builders from the Microsoft.SqlServer.Types assembly as a .NET Standard library here: https://github.com/dotMorten/Microsoft.SqlServer.Types

Would you not want the same functionality exposed via System.Spatial instead of Microsoft.SqlServer?

Note that Cosmos Db (ex Document Db) from Microsoft also has support for spatial types. Unfortunately operations on these types are to be used in LINQ expressions only and will be evaluated on server. There's no implementation provided in the client library: https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.spatial.geometryoperationextensions so it means if I'm using Cosmos Db w/o Entity Framework and w/o SQL Server (if I use Cosmos Db, it's because I don't use SQL Server...), I'm stuck.

So a standard Microsoft library (for all .NET platforms) for spatial ops would be nice.

Actually, Microsoft.Spatial is quite nice, and it has no dependencies on other assembly, but it's very tied (not technically) to "OData". I think it could just be moved into dotnet instead of staying with OData. Plus it doesn't implement simple operations like Distance (which is weird).

Would you not want the same functionality exposed via System.Spatial instead of Microsoft.SqlServer?

I want the Spatial UDT types to "just work" in and out of SQL Server. I don't care what assembly they are in.

@smourier @dotMorten @suryapratap Yes, that's why I opened this issue in the first place. There should be a single managed DLL that handles 100% of the spatial queries and evaluation. I think SQL Server should then REPLACE their spatial implementation with that one, so there is a unified, platform agnostic way to calculate and query server-side spatial data. I shouldn't have to install the SQL Spatial Types x64 DLL into my app.

There should be a single managed DLL that handles 100% of the spatial queries and evaluation

💯 agree. I'm totally happy if this is via MS or 3rd party (like how Newtonsoft Json.Net is the defacto JSON library, etc). Just need one 'main' one which is the go-to library referenced via Guidance, etc.

I think SQL Server should then REPLACE their spatial implementation with that one, so there is a unified, platform agnostic way to calculate and query server-side spatial data. I shouldn't have to install the SQL Spatial Types x64 DLL into my app.

💯 💯 💯 💯 💯 💯 The pain that this has caused me over the years :( Take the time now to 'do it right' for everyone - now is the opportunity and time.

IMHO it would be better to have the abstractions of spatial types as part of Core or even Standard, then have EF or other libs depend on it.

There are a lot of quite different ways to model geometry APIs so I'm not sure that's such a good idea either. Even a point isn't just a point with and X and Y. Sometimes it's a lat and long, or an N and E. Sometimes the have Z and M values, and the units for all of these are vastly different. And that's just the point. Lines and polygons is an entirely different way more complex story. And there are many more types of geometries used in the spatial world.

@dotMorten well, the thing is there are already some bits and pieces that Microsoft has shipped around .NET. I'm just saying these bits could be rationalized/centralized and shipped in .NET standard or in some official satellite Nuget, independently from EF, CosmosDB, SQL Server, etc.

@smourier No one is prevented from creating a netstd set of geometry classes. That doesn't mean they have to be part of the standard itself (also I'm not sure which bits and pieces of geometry you talk about that's already in the standard?).

IMHO what really needs to happen here wrt this specific issue: The SQL Server team should join the Open Source wave and opensource their UDT Types, including the native C++ bits. Since those native bits are most likely just a lot of math operations, it would be really easy to compile these for other platforms as well, and thus provide a cross-platform implementation of the class library for use on all platforms, including ios, android, Linux and mac. And I know for a fact the built-in WKT parser they have can be almost doubled in performance by switching to Span<T> and open source it would allow the community to submit such improvements. However such an implementation probably doesn't belong in CoreFX but would just as happily live in its own repo, and the existing nuget would just add a few more platform targets.

cc @divega for that suggestion.

@dotMorten - Of course, anyone can reinvent the wheel and write its own stuff, that's precisely what I want to avoid. I'm just voting for "something .NET", provided by Microsoft (not by "anyone") more or less equivalent to some of the bits they already provide (but unfortunately heavily depending on various products/frameworks/skus).

@smourier I'm curious why you say "provided by Microsoft?" What is is about an open-source non-Microsoft library that you reject?

The problem with a non Microsoft library is Microsoft things like entity framework, sql server, odata, etc... will not adopt it.

That's not true. For instance EF has dependency on a 3rd party spatial library

@ajcvickers - when I said "provided by Microsoft" I really meant provided (and used), not developed. Think about newtonsoft json. It's today central in Microsoft code (CosmosDB, ASP.NET, etc.) although it's not developed by Microsoft (well, in fact, the original author seems to be part of Microsoft now, but that's another matter :-). I'd like to see a package used in a consistent way across Microsoft products that I can use as well.

That's not true. For instance EF has dependency on a 3rd party spatial library

Yes, as @dotMorten mentioned, we have adopted a third party open source library for EF Core 2.2 spatial support.

And I have been meaning to share here an update on the path we have chosen. Although everything I have to say has already been discussed on issues in our repo, like https://github.com/aspnet/EntityFrameworkCore/issues/1100 or in our blog posts announcing preview 2 and preview 3, I think customers and contributors that have up-voted or commented on this issue may find the analysis and decisions we made relevant:

Early on, we realized that we wanted to implement spatial support in EF Core by extending each database provider with a “plug-in” package that contributes type mappings for spatial types, and LINQ query translations for some members of those types. One advantage of this approach is that it should be relatively easy to apply it to any provider, and even to any .NET spatial library.

Some time ago, our team, alongside @roji (the lead developer of Npgsql), started investigating existing spatial libraries for .NET and their capabilities. Based on the analysis (summarized at https://github.com/aspnet/EntityFrameworkCore/issues/1100#issuecomment-399571763), we moved to adopt the spatial types from an open source spatial library called NetTopologySuite (NTS for short), and the interfaces from GeoAPI for our initial implementation.

We found NTS to be the most complete, OGC-compliant library available for .NET Core today. NTS is also open source, cross platform, .NET Standard-based, database-agnostic, and developed by an active community.

So far, we have enabled this to work with SQL Server, SQLite and with our in-memory provider using NTS. Even before we started, @roji and others had already enabled similar capabilities for PostgreSQL in Npgsql, also based on NTS.

In the particular case of SQL Server support, we collaborated with the NTS community to create a package that can be used to serialize and deserialize SQL Server’s geography and geometry binary format directly to NTS types: NetTopologySuite.IO.SqlServerBytes. This allows EF Core spatial type to work with SqlClient without relying on Microsoft.SqlServer.Types.

We recognize that NTS-based solution may not satisfy everyone interested in this feature. In fact, while getting EF Core 2.2 ready for release, we have identified some opportunities for improving NTS functionality (for example support for measures – the M ordinate –, and for computing areas and distance using an appropriate coordinate system for geographic coordinates), and opportunities to improve its usability (for example, enabling more concise and clearer code for constructing spatial instances based on coordinates). But given the functionality NTS already supports, and our experience collaborating with the NTS community so far, at this point we believe it may be a better path forward for us to keep collaborating to try to make these improvements on NTS, rather than pushing for some other spatial solution, based on some other existing codebase, or from scratch.

IMHO what really needs to happen here wrt this specific issue: The SQL Server team should join the Open Source wave and opensource their UDT Types, including the native C++ bits. Since those native bits are most likely just a lot of math operations, it would be really easy to compile these for other platforms as well, and thus provide a cross-platform implementation of the class library for use on all platforms, including ios, android, Linux and mac. And I know for a fact the built-in WKT parser they have can be almost doubled in performance by switching to Span and open source it would allow the community to submit such improvements. However such an implementation probably doesn't belong in CoreFX but would just as happily live in its own repo, and the existing nuget would just add a few more platform targets.

@dotMorten I have shared your comment with the SQL Server team. I will let you know if I hear back from them about this.

This "span" improvements I hope can be easily applied to NTS as well

@DGuidi It was a pretty easy upgrade for me. Didn't take long, but make sure you use the .NET Benchmark tool to verify it's actually an improvement: https://github.com/dotMorten/Microsoft.SqlServer.Types/blob/master/src/Microsoft.SqlServer.Types/Wkt/WktReader.cs

This "span" improvements I hope can be easily applied to NTS as well

Didn't take long, but make sure you use the .NET Benchmark tool to verify it's actually an improvement

NTS, in particular, is in a weird place with things like Span<T>; because of its status as a port of JTS that we try to keep in sync with upstream JTS changes, deviating too much from what Java can do tends to be frowned upon, and I expect that if we throw in Span<T> for the sake of it, we're likely to be disappointed. Edit: that came out a bit weird... it's not that our hands are tied or anything, just that we need to have a real serious reason for deviating from JTS, since every such instance increases our maintenance cost.

I've got a really rudimentary start (edit: mainly focus on Raw*.cs, since others are optimizations of other ideas I'd had) at designing something from the ground up to take better advantage of Span<T> (namely, its elementary notion of a "geometry" is actually a ref struct wrapping a ReadOnlySpan<byte> of the baseline WKB), but I don't see much of a way forward with that approach since it can't possibly support Z or M without using something like T4 to generate the entire stack of types for each one of (XY / XYZ / XYM / XYZM). Not fun.

Even though preliminary results show that there's a ton of potential for improving really simple operations (benchmarked code for that is "Program.cs" in the same folder as that link), I suspect that the benefits will really start to disappear when it comes to implementing actually interesting algorithms like the cascaded polygon union algorithm or even the OverlayOp parts thereof, since I think those algorithms are going to just be slow by the nature of all the stuff that they have to do.

We don't plan on bringing this .NET Framework API to .NET Core. See this announcement for details.

@terrajobst is it correct to say that the way forward is to depend on NTS in applications (as per https://docs.microsoft.com/en-us/ef/core/modeling/spatial) and to consider Microsoft.SqlServer.Types as an implementation detail and/or legacy component of SQL Server?

Was this page helpful?
0 / 5 - 0 ratings

Related issues

nalywa picture nalywa  ·  3Comments

bencz picture bencz  ·  3Comments

EgorBo picture EgorBo  ·  3Comments

noahfalk picture noahfalk  ·  3Comments

jamesqo picture jamesqo  ·  3Comments