Runtime: Port over DbProviderFactories class

[FransBouma]

I have seen no info about this with respect to .NET Core, and it's a crucial element for (micro) ORMs and data-access libraries. The .NET full API for DbProviderFactory contains usage of DataTable, so I recon that's not going to be ported anytime soon, but there's a bigger problem: most provider factories are defined in machine.config, which with 'bcl per app' isn't going to fly anymore.

Have there been any design thoughts on this? Will .NET core have DbProviderFactory machinery so an ORM can obtain the factory from a central API (I couldn't find the API of .NET full in corefx at the moment), and create ADO.NET elements using that factory?

Or is this in limbo just like DataTable (https://github.com/dotnet/corefx/issues/1039)

[YoungGah]

@FransBouma, you are correct that DataTable is not the only design issue we need to solve here to make DbProviderFactory and DbProviderFactories to work. The registration mechanism for Full Framework was through config files. For NET Core, we will have to come up with a new registration mechanism, but we don't have concrete design. This will be addressed post v1.0.

[FransBouma]

Do realize that if this is introduced post v1.0, the v1.0 runtime will be a bit useless for ORMs: without a providerfactory, it's much harder to write db agnostic code that doesn't reference ADO.NET specific providers. .NET 1.x has learned a hard lesson: taking dependencies on ADO.NET providers in an ORM is a versioning nightmare. So not having _any_ providerfactory support whatsoever (not even in the app config file or whatever it's called in corefx) is a showstopper.

[davidfowl]

Just going to +1 this and say it should be all code based configuration. That's one thing I hated about those APIs in the past.

[schotime]

I have just added the DbProviderFactory as a constructor parameter now and it can be passed in when you reference your particular database driver. Eg for sql server. SqlClientFactory.Instance

It's a suitable workaround for now. I also agree with @davidfowl about a new code mechanism. However now that I think about it. You would have to register that in the app anyway so might as well just do what ive done.

[YoungGah]

@davidfowl and @schotime, yes, the current thinking is to provide code based configuration.

[eschneider999]

My micro orm Symbiotic is at %97+ to .NET Core compatibility. Just need DbProviderFactory and some config junk.

[kkurni]

Here our proposal for DbProviderFactories for corefx by using Code based Configuration
https://github.com/dotnet/corefx/issues/6476

As a background in .NET Framework DbProviderFactories is a configuration based mechanism which can be used to resolve a DbProviderFactory instance that is registered in configuration (most often in machine.config) based on a string containing a well-known “provider invariant name”. Applications using this mechanism generally don’t need to depend on the provider directly because the provider is also registered globally in the GAC.

As you probably know in CoreFx, there will be No GAC and global configuration anymore.
This make the concept of DbProviderFactories become less useful.
And for dependency injection purposes, there are lots of better alternative. e.g Contruction injection as @schotime mentioned, or using simple dictionary or your favorite dependency injection mechanism.

@FransBouma, @eschneider999, @schotime , @davidfowl or others, Please let us know whether
this DbProviderFactories will still be useful for you in Corefx context.
https://github.com/dotnet/corefx/issues/6476

[kkurni]

CC @giancarloa

[FransBouma]

I understand the lack of GAC / machine.config makes it all a bit problematic and therefore would fall back onto the mechanism of the app's config file configuration for DbProviderFactory (similar to what is possible in .NET full today), and if configuration in CoreFX is done through code, not config files then this is the wisest solution.

At first glance I see no way to obtain a factory when I have just an invariant name? As you register the factory under a name with DbProviderFactoriesConfiguration but that class stores the registered factory internally and there's no way to obtain the registered factory? Your proposal specifies contract changes, but it's unclear to me what changes where.

There's another problem, which is perhaps less of your concern, and that's factory overwriting is not supported with your solution. ADO.NET profilers (like Glimpse, EFProf and also ORM Profiler) tend to overwrite the static table in .NET full with their own factories which intercept any call to the factory instance and create wrapping elements instead to intercept calls to ADO.NET elements. Your proposed solution doesn't offer that flexibility anymore.

[kkurni]

@FransBouma you can access your registered factory the same ways as the old ways in .NET Frameworks.

DbFactories.GetFactory(string providerInvariantName);
DbFactories.GetFactory(DbConnection connection);

In CoreFx, you will need to use DbProviderFactoriesConfiguration to register your provider.

DbProviderFactoriesConfiguration.Add("Sytem.Data.SqlClient", () => SqlClientFactory.Instance);

But in .NET Frameworks, you don't need to do anything, it will read from the machine config as it is.

[kkurni]

The reason why we are not allowing to overwrite is to make the behavior less uncertain.
e.g libraries can just overwrite this code based config without any agreement. This will be hard to debug.

But an applications may be able to provide their own safe way to overwriting their provider config.
e.g file based config which allow others to override their config providers based on directory structure.

[FransBouma]

Re: retrieving the factory: ah yes, it was unclear a bit to me whether that was changed or not, this morning I realized it was indeed the same. Sounds good!

Re: overwriting: ok. It's not a big deal I think. The applications now also have to call a line in the profiler to make it overwrite the factories in the datatable, and instead of that the developer can simply register a different factory (i.e. the one from the profiler instead of the regular one). It's a bit more explicit but at least it is possible to do.

Regarding dependencies: I think the main change is: in regular .NET, an application doesn't have to reference an ADO.NET provider assembly directly, and in CoreFX this will be different: The application has to reference the ADO.NET provider assembly directly as the registration of the factory (which is in the ADO.NET provider assembly) has to be registered. I think this unavoidable as there's no central GAC/machine.config so it is what it is.

So IMHO it all looks OK to me: porting code over to CoreFX will now be easier as the mechanisms are the same. Thanks for this!

[davidfowl]

Here's a question for you @FransBouma, how are you using this API? Since there's no config system and the application is responsible for wiring everything up anyways, why not just provide an API that takes a factory directly. It's up to user code to wire it up

[FransBouma]

Here's a question for you @FransBouma https://github.com/FransBouma , how are you using this API?

What do you mean with this question? As there's hardly any API to use: you call the static class' method to obtain the factory for a known name, you get the factory, cache it (as it's a static instance), and use the instance to create ADO.NET objects for using with a database.

At least that's how I (and everyone else) use it on full .NET. On CoreFX I haven't used it yet, as it's impossible to create proper software with the pre-RC2 tooling.

Since there's no config system and the application is responsible for wiring everything up anyways,
why not just provide an API that takes a factory directly. It's up to user code to wire it up

No, that's way too simplistic. The application wires things up for the application, but it has not and should not know internals about the used ORM / Data access layer, other than perhaps which ADO.NET provider to use, but not how to set things up internally with that. Besides, it's not always known to the application when the ORM is used, or which ORM, or which database up front, only at runtime.

It also makes things overly complicated as current code can now easily be ported to CoreFX because there's no direct line between application logic and the query engine of an ORM: it obtains the factory through the system. Your proposal requires code to be refactored because the factory has to be supplied directly to the query engine of the ORM by the application logic.

In all honesty I then wonder what the real point of your question is: is this API being cut?

[kkurni]

@FransBouma sorry to put high expectation for this proposal initially.

At this moment this proposal is currently under reviewed by internal teams which raised couple concern.
The biggest concern at this moment is "this DbProviderFactoriesConfig will caused confusion for PCL Libraries". e.g if a PCL libraries register explicitly through DbProviderFactoriesConfig and bring this PCL to .Net Framework (e.g Asp.net), all those registration will get ignored. and .Net Framework will use Machine.config instead of whatever that had been registered in those PCL Library.

And other concern is about "the original reason why we don't bring this DbProviderFactories in CoreFx".
We try to clean the new CoreFx API as possible from anti pattern. and we think without this DbProviderFactories, library writer can still depend directly from DbProviderFactory. the application writer can provide / inject this DbProviderFactory directly through the library and the library will just use DbProviderFactory directly.
At the end of the day, this actual implementation of DbProviderFactory (e.g SqlFactory) will need to be registered anyway from the application with this proposal.
(The library/ORM,still can depend on DbProviderFactory - System.Data.Common, but the application who use that library will depend on SqlDataProvider even with this proposal)

So at this moment, we need to gather more information from the community for the usage and suggest alternative way to use DbProviderFactory directly. We still need to do more analysis and threat modeling for this proposal.

I will keep you up to date if there is any progress on this and please give us any input and suggestions

Thanks

[FransBouma]

At this moment this proposal is currently under reviewed by internal teams which raised couple concern. The biggest concern at this moment is "this DbProviderFactoriesConfig will caused confusion for PCL Libraries". e.g if a PCL libraries register explicitly through DbProviderFactoriesConfig and bring this PCL to .Net Framework (e.g Asp.net), all those registration will get ignored. and .Net Framework will use Machine.config instead of whatever that had been registered in those PCL Library.

Yeah, but that's a problem you have regardless: if you create a system to obtain the factory through methodology ABC in PCLs, it won't match what's done in normal .NET.

If ADO.NET using code can't utilize DbProviderFactory on CoreFX, porting costs will become big. The last thing I want is my SQL engines having a dependency on the ADO.NET provider they indirectly are using. That sucked big time during .NET 1.x as every time the ADO.NET provider versioned, the ORM had to version as well, plus supporting multiple ADO.NET providers through one SQL engine (e.g. Oracle with ODP.NET with both a managed and a CLI wrapping ADO.NET provider, with at least 3 versions shipped across 2 CLR versions) is then out of the question and it's trivial to do today.

I really don't care _how_ the DbProviderFactory is registered, as long as ADO.NET using code (e.g. the SQL engines which create connections, DbCommand instances and the like) can obtain the DbProviderFactory through a central class by supplying a known name.

If the application has to register the factory with the ORM, things get limited and you put the burden of how things are registered onto us ORM writers and our users instead of solving it yourselves properly.

Anyway, I'm so tired debating trivial ADO.NET stuff with Microsoft time and time again. Do what you have to do, cut it, don't cut it, I don't really give a shit anymore.

[FransBouma]

The main issue is: the ORM is perhaps used 3 layers deep. The application at start point doesn't have the notion of that ORM _today_. If the application has to do the registration of the factory with the ORM (which is now abstracted away!), this means that it has to go through these 3, 4 layers to reach the ORM to register the factory, a coupling that doesn't exist today with the DbProviderFactory system.

Not only that, but the ORM has to be redesigned to make it possible to specify the factory through the outside, something that has to be changed with the current designs which simply obtain the factory from the system.

This is all major pain and misery and additional burden for code to port to CoreFX. If it's your goal to make porting existing frameworks to CoreFX a miserable experience, you have your work cut out for you.

Seriously, make it as easy as possible to port code, that's all we ask. It benefits us, but also you, as the last thing MS needs is lack of people willing to port code to CoreFX.

Cheers.

[eschneider999]

Maybe you should talk to the Entity Framework team, don't they use the dbproviderfactory also?

-----Original Message-----
From: "Frans Bouma" [email protected]
Sent: ‎3/‎29/‎2016 3:02 PM
To: "dotnet/corefx" [email protected]
Cc: "eschneider999" eric.[email protected]
Subject: Re: [dotnet/corefx] What's the state of DbProviderFactory? (#4571)

The main issue is: the ORM is perhaps used 3 layers deep. The application at start point doesn't have the notion of that ORM _today_. If the application has to do the registration of the factory with the ORM (which is now abstracted away!), this means that it has to go through these 3, 4 layers to reach the ORM to register the factory, a coupling that doesn't exist today with the DbProviderFactory system.

Not only that, but the ORM has to be redesigned to make it possible to specify the factory through the outside, something that has to be changed with the current designs which simply obtain the factory from the system.

This is all major pain and misery and additional burden for code to port to CoreFX. If it's your goal to make porting existing frameworks to CoreFX a miserable experience, you have your work cut out for you.

Seriously, make it as easy as possible to port code, that's all we ask. It benefits us, but also you, as the last thing MS needs is lack of people willing to port code to CoreFX.

Cheers.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dotnet/corefx/issues/4571#issuecomment-203075152

[kkurni]

Thanks @FransBouma for sharing your pain. No, we don't cut this API yet.
It is still under review to provide best possible solutions. I just share our concerning part which came out from internal review.
After talking to @terrajobst, we find a solution with versioning issue in PCL so that it will be less confusing. I will update the proposal soon.

@eschneider999 Yes, we already talked to EF Team and they have their own container / configuration mechanism.

   optionsBuilder.UseSqlServer(connectionString);

so they don't use DbProviderFactories (Plurals) but they still use DbProviderFactory (Singular) for abstraction layer. DbProviderFactory (singular) is available in CoreFx since the beginning.

[eschneider999]

Lol, there's the solution and they didn't even know it....

-----Original Message-----
From: "Kurniawan Kurniawan" [email protected]
Sent: ‎3/‎29/‎2016 5:18 PM
To: "dotnet/corefx" [email protected]
Cc: "eschneider999" eric.[email protected]
Subject: Re: [dotnet/corefx] What's the state of DbProviderFactory? (#4571)

Thanks @FransBouma for sharing your pain. No, we don't cut this API.
It is still under review to provide best possible solutions. I just share the concerning part that we have right now which comes out from the review.
After talking to @terrajobst, we find a solution with versioning issue in PCL so that it will be less confusing. I will update the proposal soon.

@eschneider999 Yes, we already talked to EF Team and they have their own configuration. so they don't use DbProviderFactories (Plurals) but they still use DbProviderFactory (Singular) for abstraction layer. DbProviderFactory (singular) is available in CoreFx since the beginning.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dotnet/corefx/issues/4571#issuecomment-203136012

[terrajobst]

As @kkurni said, we believe we've a design that satisfies the request. Here are the goals as I understood it by @FransBouma:

• Expose DbProviderFactories for libraries so that components (such as an ORM) can ask for a specific DbProviderFactory by simply passing in a string they know via _some_ means (e.g. configuration).
• On .NET Framework, the built-in providers are automatically available via the machine.config. Application authors can _additionally_ register others by adding them to app.config
• On .NET Core, application authors are expected to register _all_ providers by calling a to-be-added API.
• Long term, our plan is to support this to-be-added API on both, .NET Framework as well as .NET Core, so that code registering providers could be portable as well. However, the lack of this API on .NET Framework isn't a blocker for today's scenarios.

Specifically, that's the API shape for .NET Core as I discussed it with @kkurni:

``` C#
namespace System.Data.Common
{
public static class DbProviderFactories
{
// V1 -- Supported by .NET Framework 4.6.2 (the upcoming version)
public static DbProviderFactory GetFactory(string providerInvariantName);
public static DbProviderFactory GetFactory(DbConnection connection);

    // V2 -- Supported by .NET Core 1.0 and by .NET Framework vNext (version after 4.6.2)
    public static void RegisterFactory(string providerInvariantName, DbProviderFactory factory);
    public static IEnumerable<string> GetFactoryProviderNames();
}

}
```

Is that accurate?

[FransBouma]

Is that accurate

Yes. Application startup on CoreFX registers factories to use by application. Somewhere deep in the application's call chain, code can now ask the system for a factory using a well known name.

In general these names are known up front. So e.g. the Oracle engine uses name X, and whatever is registered under X, be it the managed provider v12c, or the unmanaged v9i, it works.

So for CoreFX all that's required is extra machinery to register a factory which is done on .net Full through machine.config / app.config, which then used by the application's startup code. Everything else between that and the low level dungeon of the data-access code is the same: DbProviderFactories.GetFactory is used to obtain a factory, providing a name X.

Application architecture

Start -> A -> B -> C -> D -> | E -> F -> ADO.NET -> Database

In the situation above, A..F are components/layers which offer some form of abstraction in the application and together form the logic of the application. E & F are the ORM, where F is the sql engine actually using the factory.

.NET full

In .NET full, the factory is registered in machine.config or app/web.config, and nothing is done further: F requests factory from system and uses that to use ADO.NET classes.

CoreFX, this proposal

In CoreFX, the most ideal way to make this work is to do registration in A. This would mean all other code further down the road (B->F) can be the same as on .NET full: in F on CoreFX, the same code is used to obtain the factory to use ADO.NET classes.

IMHO this also means that CoreFX specific code is located in A (and that's fine, it's a CoreFX app!), and all other code (B->F) is not relying on CoreFX specific APIs: they use the same API on .NET full as on CoreFX: Only F has to obtain the factory and does so using an API that's shared between .NET full and CoreFX.

[davidfowl]

So this is all about compatibility and nothing more (Like a better API design?). If so, then it looks fine to me.

[FransBouma]

So this is all about compatibility and nothing more (Like a better API design?)

With all due respect, 'better' is highly subjective here. In my schematic overview in my previous post, the DbProviderFactory registered with the container that runs the application is the best solution, as it gives the least amount of friction and problems. Your proposed solution requires a public API on B, C, D, E _and_ F to accept a DbProviderFactory instance, which leaks the abstraction of F all the way up to A. There are many other downsides to this, which I hope I don't have to illustrate.

Connecting to a database is a field where a lot of people have come up with a lot of different ways to do just that, with all the different configuration and setup crap they could possibly think of, from tsnnames.ora to odbc to oledb to connection strings in config files etc. On .NET we have a reliable way to work with all of that in a standardized fashion which works since .NET 2.0 RTM (and if you want, you can use the .NET 1.x variant as well, the direct reference to the ADO.NET provider, bypassing factories altogether) and has given us highly decoupled frameworks from the actual connection/setup crap that has to be there regardless of what .NET forces you to use.

Changing that doesn't solve _any_ problem.

[eschneider999]

Frans,

So i'm trying to understand the need for the string that is passed to the factory? is it to indicate a specific database/type? The specific instance should be doable in the connection string.

All I need is an abstraction layer so I don't have to bind against specific database libs. I have that now (custom) but would prefer to use some standardize method.

How is the config/registering of database client libs to be handled in core?

Why can't/should we just use the same factory as the Entity Framework team?

[FransBouma]

So i'm trying to understand the need for the string that is passed to the factory? is it to indicate a specific database/type? The specific instance should be doable in the connection string.

Please read the MSDN articles regarding DbProviderFactory.

All I need is an abstraction layer so I don't have to bind against specific database libs. I have that now (custom) but would prefer to use some standardize method.

DbProviderFactory solves precisely that. You do:

var factory = DbProviderFactories.GetFactory("Oracle.DataAccess.Client");

and if the factory for ODP.NET unmanaged is installed on the system (which is done when you install ODP.NET), you'll get the factory back. If not, you'll get an error. Which version of ODP.NET is used is not important: this can be 9i's 10gs, or 12c's for example. So your code is independent of the ADO.NET provider version. You'll get the factory of the version which is installed (if it's installed ;)). The factory can then be used to create connections, commands etc.

How is the config/registering of database client libs to be handled in core?

see proposal above earlier in this thread.

Why can't/should we just use the same factory as the Entity Framework team?

EF uses DbproviderFactory as well and why should anyone take a dependency on EF's code if what they do has nothing to do with EF? this is about using ADO.NET, not about EF.

[eschneider999]

Ok, so the string is the name of the db client lib. Been a while since I have been in there..

So the EF factory is in EF specific lib? If so I agree we don't want to use that, but if that's the case couldn't we just move it to a core lib?

What is the proposed core registration process? a config file or some other means? Is the registration specific to an app or a machine wide registering of the database libs?

Anyway, other than some details, I agree with your proposal...

[terrajobst]

So the EF factory is in EF specific lib? If so I agree we don't want to use that, but if that's the case couldn't we just move it to a core lib?

While I'm not opposed to to doing this, I don't think it would add a lot of value. In general, we usually prefer existing APIs that do the job well over newer APIs just because _insert fashion argument here_.

[davidfowl]

I don't have a stake in this, I'm mostly asking questions to learn more.

and if the factory for ODP.NET unmanaged is installed on the system (which is done when you install ODP.NET), you'll get the factory back. If not, you'll get an error. Which version of ODP.NET is used is not important: this can be 9i's 10gs, or 12c's for example. So your code is independent of the ADO.NET provider version. You'll get the factory of the version which is installed (if it's installed ;)). The factory can then be used to create connections, commands etc.

How would this work on .NET Core. There is no global configuration system, or GAC. I'm just not connecting the dots here. The API proposed is a glorified dictionary. Is the argument that it's a dictionary in the System.Data namespace so everyone will use it?

[FransBouma]

The current API in .NET full is a glorified datatable ;) anything that can provide me at runtime with an instance of the factory for an ADO.NET provider works. DataTable isn't suitable for core, a dictionary can work fine too. It's not a complicated system: you have a unique name and it resolves through the API to a type that's instantiated and the instance is used to create objects on the fly for ADO.NET: parameters, commands, connections etc. So name-type pairs is all it contains and the API returns a type (or instance) based on the name you provide.

In .NET full you have the config system in machine.config, in .NET core that's absent so this api here proposes a simple registration system for specifying the name-type pairs to use so generic code which only has the name and not the type (as it doesn't reference the ADO.NET provider) can obtain the factory based on the name it has.

[grahamehorner]

DbFactories should also support the creation of the provider {Provider}Dependency object; and the {Provider}Dependency should implement a common interface IDbDependency to allow abstraction and/or generic implementations.

[FransBouma]

@karelz Considering the removal of the label 1.2.0, I recon this feature won't make it in netstandard20? Is there an alternative implemented for .net core so DbProviderFactory classes can be registered centrally? There was some debate about that but I lost track whether that was actually checked in or lost at the table of brilliant ideas...

[karelz]

@FransBouma you are correct that Future means, not in .NET Core 2.0 (which implies not in .NET Standard 2.0).
@saurabh500 @YoungGah can you please answer second part of the question above?

[karelz]

@saurabh500 @YoungGah ping?

[chrisfcarroll]

I followed the thread and I'm still lost, but I think the current state is, there is no equivalent to DbProviderFactories in Core at the moment. Have I followed correctly?

[danmosemsft]

@saurabh500 @divega could you please answer the question above?

[divega]

@saurabh500 I am not sure what is the current plan for DbProviderFactories in 2.0, can you provide some details?

I understand we were originally looking at having a new .NET Core-specific provider registration mechanism but then we switched to trying to get DbProviderFactories in. I also remember seeing a brief discussion between you and @terrajobst about whether DbProviderFactories in .NET Core could use System.Configuration or have its own registration API, but I cannot find it at the moment.

[karelz]

@divega I think you mean https://github.com/dotnet/standard/issues/254

[jemiller0]

I have some database code that uses DbProviderFactories to create database connections in a DBMS independent fashion. Can someone please tell me what the recommended way is to get this code to work in .NET Core/.NET Standard? How does this work in JDBC? As far as I know, Java/JDBC doesn't have a configuration file where the providers need to be registered. Maybe it uses reflections to look for static strings in classes or something? Honestly, I'm amazed that something like this could not make it into .NET Standard. I've always wondered why Microsoft's documentation examples are always using the SQL Server specific classes. And the generic classes were always annoying much more imperative where you needed to use tons of statements to get anything done. I really think something needs to be done to improve this from a usability standpoint. I don't know why this has to be so complicated. Somehow, the Java people figured it out decades ago. If I remember correctly, the provider name is part of the connection string. And please, do get rid of dependence on DataTable. The metadata APIs are completely useless. You can really tell that providing a DBMS independent API was an afterthought. And now, it appears to once again be overlooked. How could we be at 2.0 and something this basic could not be supported?

[terrajobst]

A few comments:

• DbProviderFactory is part of .NET Standard 2.0. It's correct that DbProviderFactories doesn't exist in .NET Standard. The reason being that we don't have a good way to hook up providers without adding them to the implementation. We currently don't have a great way to inject configuration -- Xamarin have removed System.Configuration because the implementation (necessarily) has dependencies on all configurable technologies, thus making tree shaking in static linking scenarios nonviable.

• SQL Server isn't special. It's true that it's used more in our documentation, but that has more to do with heritage than being a recommendation or hidden marketing play. Note that the SQL Server provider isn't part of .NET Standard either.

The goal for .NET Core 2.0/.NET Standard 2.0 is mostly establishing a useful compatibility baseline with .NET Framework to help customers bring in existing code. Changing API interactions in core APIs, like removing DataTable from the metadata discovery, isn't helpful. Of course, this doesn't prevent us from adding better APIs in the future.

[divega]

@terrajobst FYI, I updated the title of the issue to reflect that this is about DbProviderFactories (plural) rather than DbProviderFactory (singular). I think it might help avoid confusion.

[eschneider999]

There is a common database api System.Data.Common, which can be found on Nuget.

As for the DBfactory stuff I rolled my own for my ORM

http://www.nuget.org/packages/Symbiotic_Core_x64/

[FransBouma]

@terrajobst There's currently also no standard way to pass a DbProviderFactory to the application as well. I think @jemiller0 has a point wrt Sql Server: as its ADO.NET provider is in the BCL, there's no real pressure from MS to design a real API to pass the DbProviderFactory instance to the application code in a generic fashion. I don't really see how that's difficult, as all DbProviderFactory types derived from that class, so all it takes is a structure that's kept in memory for the duration of the application, which is filled by a standard extension method to pass a DbProviderFactory to the structure and which offers a standard API to ADO.NET using classes to pull a factory from that structure.

You can't sell me the blurp that that is so hard to design, it's simply not _needed_ for MS as SQL Server is reachable inside the BCL. Now all ORMs have to roll their own API, own structure which is kept in memory, own way to obtain that and own methods to register providers with that structure.

Why this is still a problem 1.5 years after I first asked this question is beyond me, considering there are solutions available on competing platforms.

[FransBouma]

IMHO it would be best if the complete ADO.NET class space would be moved to a team which has no stake in either EF nor SQL Server, like it was before in the early/ier days. The main benefit with that is that it forces that team to design and maintain their API with external consumers in mind, not themselves (as is the case now).

Now we have to hope that the SQL Server team is generous enough to design an API that benefits others besides themselves. I think in the past 2 years they've proven not to be up to the task.

[divega]

There's currently also no standard way to pass a DbProviderFactory to the application as well

@FransBouma can you please elaborate a bit more on what you mean by this? I want to understand the rest of your post but I am stuck at this.

[jemiller0]

Since DbProviderFactory is supported, I think that will be good enough for my needs. I'm thinking I can do something along the lines of the following, although I haven't had a chance to test it.

DbProviderFactory dpf = dbProvider == DbProvider.SqlServer ? SqlClientFactory.Instance : dbProvider == DbProvider.MySql ? MySqlClientFactory.Instance : dbProvider == DbProvider.PostgreSql ? (DbProviderFactory)NpgsqlFactory.Instance : throw new NotSupportedException();

Code after that would simply use the DbProviderFactory interface to work with the providers in a generic way.

In my case, it's fine that the provider DLLs are included with the library that I have. Maybe if I was creating an ORM that needed to support any provider and I needed to do things in a more dynamic fashion I would have issues.

I know that NHibernate needs updating to be able to run on .NET Core. And I use NHibernate for some things. So, I hope the NHibernate developers are able to do what they need to do to get it to work on .NET Core/.NET Standard. I saw on their site that lack of DbProviderFactories was one of the things that was holding them up.

[FransBouma]

@divega

can you please elaborate a bit more on what you mean by this? I want to understand the rest of your post but I am stuck at this.

ORMs have generic code, and aren't hard-referencing the ADO.NET provider. On .net full, they obtain the ADO.NET DbProviderFactory by calling the GetFactory() method on the DbProviderFactories class. This way nothing is hard-referencing ADO.NET providers.

On .NET core, there's no machine.config nor a GAC so the application has to register the DbProviderFactory class type somewhere so the generic ORM code knows what DbProviderFactory to use. Currently there thus has to be used a custom method supplied by the ORM where the DbProviderFactory type is passed to the ORM somehow at app startup so the ORM knows which DbProviderFactory to use. I propose that there's a generic method supplied which registers a DbProviderFactory somewhere so the ORM can obtain the DbProviderFactory type based on a name, like before, without the necessity of a per-ORM supplied method which registers the factory with the provider directly.

[divega]

@FransBouma thanks for clarifying.

Currently there thus has to be used a custom method supplied by the ORM where the DbProviderFactory type is passed to the ORM somehow at app startup so the ORM knows which DbProviderFactory to use. I propose that there's a generic method supplied which registers a DbProviderFactory somewhere

I think this is a good summary of what is missing right now and in my opinion the main motivation to keep this issue open. DbProviderFactories would probably be able to fill that role again, but it needs a standard way to register providers in code, rather than only through some external configuration. And by "standard" I mean that it would really be great if that mechanism was in .NET Standard and implemented widely.

[KevalPatel]

Is there any workaround to make it below line work ?

c# var factory = DbProviderFactories.GetFactory("System.Data.SqlClient"); var conn = factory.CreateConnection();

[Nicholi]

@KevalPatel

var factory = SqlClientFactory.Instance;
var conn = factory.CreateConnection();

After almost two years that is the "solution". Essentially you have to handle getting the correct DbProviderFactory yourself, whether it be SqlServer or MySql. Sorta sad this isn't even going to be in netstandard2.0.

[karelz]

@Nicholi which API would you expect in .NET Standard 2.0? (note that only .NET Framework 4.6.1 APIs which are also on Xmanarin & .NET Core are allowed to be in .NET Standard -- if .NET Core is the only platform missing something, I would like to know)

Or did you mean existence in .NET Core 2.0 as the first platform to lead the way ahead of other .NET platforms? (I could understand that)

[Nicholi]

@karelz This API https://github.com/dotnet/corefx/issues/6476
```c#
namespace System.Data.Common
{
public static class DbProviderFactories
{
// V1 -- Supported by .NET Framework 4.6.2 (the upcoming version)
public static DbProviderFactory GetFactory(string providerInvariantName);
public static DbProviderFactory GetFactory(DbConnection connection);

    // V2 -- Supported by .NET Core 1.0 and by .NET Framework vNext (version after 4.6.2)
    public static void RegisterFactory(string providerInvariantName, DbProviderFactory factory);
    public static IEnumerable<string> GetFactoryProviderNames();
}

}
```

I don't really expect anything now, it's clearly not happening.

[FransBouma]

@karelz any API, it's taken a LONG time before Microsoft even acknowledges that there should be 'an' API for this (like, a year?) and now there isn't any.

We ORM devs can write our own, and write something that will 'work', but it sucks that something so crucial for the whole ADO.NET ecosystem is so overlooked. You know what that tells me? That the people who have controlled System.Data for so long have no clue what on earth they're doing, as otherwise they would have solved this little problem eons ago (it's not as if this is a new problem just brought to light!)

[karelz]

@FransBouma I understand your frustration, but I would like to ask you to tone down your message and keep it constructive. I am trying to help here!
However, personal-ish / team-ish attacks are not likely going to help with easier communication with the team you want something from. If I were them, I would be scared now to even talk to you or ask you for advice - just to avoid further attacks. I assume it is not an outcome you want, right?

[karelz]

@jemiller0 you are turning the technical discussion into much larger "I don't like your general direction across all the stack" discussion.
That is fair feedback and I am happy to discuss it, but it is off-topic here. If you're interested, please file a new issue (and tag me) and let's have the discussion there (forums may be even better place, but this will do for now) -- I won't attempt to answer any of your particular questions here to avoid derailing the discussion about DbProviderFactories. Thanks for understanding!

[jemiller0]

@karelz Sorry about that. You're right.

[karelz]

@jemiller0 thanks for understanding! As I said, happy to discuss particular things elsewhere - just tag me ;-)

[FransBouma]

I understand your frustration, but I would like to ask you to tone down your message and keep it constructive. I am trying to help here! However, personal-ish / team-ish attacks are not likely going to help with easier communication with the team you want something from.

I know you mean well, but coming into this thread with 'which API would you expect in .NET Standard 2.0?' as you did shows you either haven't read all posts (understandable, it's very long) or everything that's said in this thread has been for nothing as we're still at square one. I truly hope we are NOT, but it seems we are.

Sorry, but that frustrates me to no end. I've been dealing with ADO.NET for .NET core mess for a long time now, this issue as well as e.g. the DataTable issue has dragged on a long time with no real feedback coming from the team (whoever they are!) or any constructive design coming from Microsoft other than dragging feet.

If I were them, I would be scared now to even talk to you or ask you for advice - just to avoid further attacks. I assume it is not an outcome you want, right?

Oh come on... Microsoft releases not one but two ORMs for free, competing with my work (which is my source of income, mind you), installs them on every computer, tells every developer there's only one way to deal with data and that's with MS' ORMs. All that with a budget of > $1M a year for these frameworks (team of at least 10 people) and 0.0 direct revenue (as they're free!). This has resulted in Microsoft has pushed almost all commercial and non-commercial ORMs into a niche corner or out of business.

That's the situation we ORM devs are in. So if I am a little frustrated with Microsoft's dealings with respect to an API we all need, then you know why that is, besides Microsoft not showing any intend to really solve the matter other than restarting the same discussion every few months. Because that's what's happening: the debate is started, people chime in, MS personnel joins, does some proposals, things go quiet and we all learn the MS people have moved on to other things, effectively leaving this discussion hanging in the void. It's then restarted, same things happen again, rinse repeat.

I understand 'the team' (I have no idea who they are, I have never met anyone from that team on GitHub) doesn't like to be told they are dropping the ball in places, but frankly with all that's happened, I don't really care anymore.

Again, not your fault, and it's nothing personal, it's a state of frustration that is kept in place by your employer and its tactics: we outsiders have to fight a very steep uphill battle for features we need but you don't. At least that's how it feels. And your post about 'what API should there be?' triggered it again, simply because it marked the start of a new cycle of discussions about the same topic till that discussion falls flat, the Microsoft people involved move on and after a quiet period it will be restarted again with likely the same question.

This thread in particular is very frustrating as there was a very solid proposal by a MS employee earlier on, which had everything that's needed (as this is really a thing an intern can write in an afternoon, it's nothing difficult about it: a concurrent dictionary with a method to register things in it, that's it!), but apparently no-one from Microsoft to be able to read that.

I've been trying to be friendly in this thread and in others, but sometimes things frustrate too much and friendliness apparently only gets you a discussion that 'indeed things have to be done', 'it should be solved', 'good point' etc. and it doesn't get anywhere after that.

So it gets to a point, after more than 1.5 years, that it either should be solved or closed. Just implement the API, solve it, move on, or close it as 'won't fix'. It has dragged on long enough. Yes I know the teams are all in crunch mode chasing a self-inflicted deadline but you have to understand: netstandard 2.0 is a very important milestone for 3rd parties: they will start porting code big time to netstandard 2.0 when it comes out. If this API isn't in there, people have to find ways to solve it themselves, and after that there's no use for adding it anymore, as everyone has a solution themselves already.

[MihaMarkic]

That's the situation we ORM devs are in. So if I am a little frustrated with Microsoft's dealings with respect to an API we all need, then you know why that is, besides Microsoft not showing any intend to really solve the matter other than restarting the same discussion every few months. Because that's what's happening: the debate is started, people chime in, MS personnel joins, does some proposals, things go quiet and we all learn the MS people have moved on to other things, effectively leaving this discussion hanging in the void. It's then restarted, same things happen again, rinse repeat.

I can certainly confirm this unfortunate pattern. As a long time ADO.NET user (like from .net beta times) I'm also indirectly affected with it and I can't understand it. Can't somebody actually listen and once for all solve the problem?

[roji]

As the owner of Npgsql (ADO.NET open source driver for PostgreSQL), I have to agree with @FransBouma's comments... while the EF team communicates with the community, has relatively clear roadmaps and is reasonably reactive etc., ADO.NET is in a pretty sorry state - very little idea of who works on it, what long terms plans are, and à general impression that it's sort of abandoned. I can point to several issues that have hardly received any attention over the years, it can be quite frustrating.

[eschneider999]

I agree with all the other ORM guys, yet seems we all worked around the problem. I just made an interface wrapper and provided specific libs for each vender, I guess I could have used reflection but wanted something quick and easy. I would imagine we could provide a nuget package ourselves…

I guess I would rather concentrate on TransactionScope support, another ball Microsoft keeps dropping.

[divega]

Everyone,

First of all, I want to be fair to the team that has been handling ADO.NET issues: they actually care about problems like this, and they started 2.0 with the intention to address them. We even had design discussions about this particular class. Further in 2.0 the direction of the project changed, DataSet was brought into the picture and they had to make new plans. In general they have been busy, making other basic scenarios work correctly across multiple platforms, and also having SqlClient work against multiple editions of SQL Server on all platforms, which they had to prioritize.

In retrospective, we overall have not been investing enough in ADO.NET. This is also an area in which the participation of the community has been concentrated on discussions, but I believe we probably haven't done enough to enable the community to help address the issues by contributing their code. We really want to fix all of that, and we are working on a plan for it. But to set expectations, very little will change before 2.0 goes RTM.

[jemiller0]

I just try to think about it as a complete re-write of .NET. That's not something that is going to happen overnight. It would be great if after other higher priority issues were addressed, that something could be done to modernize the ADO.NET Core API (I think that's what it was called, long before everything else was called Core). Also, it would be nice if the name ADO could be dropped. ActiveX Data Objects? Please, let us all forget ActiveX.

[eschneider999]

Maybe Microsoft can stop working on the Visual Studio start screen and work on this?

Honestly Microsoft priorities are crap, I never use a dataset, anybody who knows what they are doing uses an ORM and probably not Microsoft’s.

They can’t even fix simple things everyone wants:
https://connect.microsoft.com/SQLServer/Feedback/Details/339410

I mean that’s 9 years old…

[jemiller0]

Yeah, there are definitely usability issues like that that seem small, but, could make working with it much easier. I've noticed a lot of things with the SQL Server provider, such as having to specify different parameters such as lengths that other providers don't require and seemed nonsensical. Things like having to specify a length for a DateTime parameter and other oddities. Or having to specify lengths when you are using a prepared command multiple times for a string field. I kind of wondered if anyone was even working on ADO.NET Core anymore. As far as I can tell, not much has changed with it since .NET 1.0. At least with regard to the API.

[FransBouma]

I will work on this today and will push commits to dotnet/corefx#19826, as that PR was added solely as ground work for this class.

As corefx doesn't have machine.config nor other config files, it won't auto-initialize itself with config data: factories have to be registered by the developer. I'll add two methods to register a factory:
one which accepts a name and a DbConnection derived instance and one which accepts a name and a DbProviderFactory derived type. I picked those two as I think either one of them is easy to use for a group of users: either they know the type of the connection class to use, or they know more about factories and are familiar with DbProviderFactory derived classes and know the type to use.

I'll use a ReaderWriterLockSlim to lock the internal structure as registration of a factory and a read for a factory instance can be done on multiple threads and can cause a race condition. I'll still use a DataTable structure internally, as that will keep the functionality compatible with .net full where the datatable is overwritten by profilers like e.g. Glimpse, Hibernating Rhino's profilers and my own ORM Profiler with wrapping factories.

I'll auto-initialize the data structure with the SqlClientFactory instance through reflection, to avoid a cycle in references and to make life easier for users, as they then don't need to register the factory with the DbProviderFactories class. ADO.NET provider writers can decide to autoregister the factory internally in their code by e.g. adding a static ctor to their DbConnection derived class and register the factory there. A bit of a hack, but would make things completely transparent to the user.

(edit) I see Odbc is also present in CoreFX, will add that factory automatically as well.

(this is in line with the current .net full implementation, where it initializes itself with 4 factories which are in .net full distribution)

[jnm2]

I'm looking forward to see what you come up with because on .NET Framework, I've had to use reflection and directly modify the backing DataTable to dynamically register at runtime.

[FransBouma]

at .net full you can register the factory either in your application's app/web.config file or in the machine.config file, there's no need to alter the backing datatable at runtime on .net full (other than to wrap existing factory registrations).

[jnm2]

@FransBouma Specifically for scenarios where I can't or don't want to register the factory ahead of time in a .config.

[FransBouma]

the design of the system on .net full is based on configuration of factories ahead of time, I have never ran into a scenario where it wasn't appropriate to have the factory registered ahead of time. The factory is read once anyway, so registering it later on dynamically doesn't save you anything, and in fact could run into race conditions as the table you're altering isn't guarded by a lock. Anyway, that's for net full and this is for .net core :)

[jnm2]

@FransBouma This is two unit testing factories. Managing .config files is too much overhead for lots of test dlls. It should be plug and play. Anyhow, whatever you do in corefx will be interesting. I'll see if it can be polyfilled on netfx.

[divega]

@FransBouma FYI, I went ahead and created an issue to bring this API into .NET Standard: https://github.com/dotnet/standard/issues/356. The exact version of .NET Standard is going to depend on schedule. The proposal reflects some thinking I have done on the API, but it is completely open to feedback. A few principles:

1. We should have existing API in .NET Framework and Mono as the baseline. That includes the DataRow/DataTable APIs for compatibility.
2. We need to enable in-memory and hybrid implementations.
3. I used SetFactory for the in-memory registration because is the opposite to GetFactory and I wanted the API to succeed always regardless of whether a provider was previously registered with the same invariant name.

Besides that, I think in the .NET Core implementation we need to give careful consideration to thread-safety and the possibility that the provider registrations change at runtime.

cc @terrajobst

[FransBouma]

@divega

netstandard should be great indeed, otherwise people who create a netstandard targeting dll can't see the class anyway (as I'm currently facing in the tests as I can't add a test, so I can't proceed at the moment :( , but that's discussed in the PR)

We should have existing API in .NET Framework and Mono as the baseline. That includes the DataRow/DataTable APIs for compatibility.

The API indeed does this for 99.9%. In the .NET full version, it registers 4 factories, always: ODBC, OleDB, SQLClient and OracleClient. On .NET core, OleDb and OracleClient are absent, so I decided not to preserve these 2 rows in the datatable (they have predefined indexes in .net full, why I have no idea) as it would leave 2 empty rows in the table when GetFactoryClasses() is called.

We need to enable in-memory and hybrid implementations.

Not sure I understand what you mean by this, could you elaborate a bit on this?

I used SetFactory for the in-memory registration because is the opposite to GetFactory and I wanted the API to succeed always regardless of whether a provider was previously registered with the same invariant name.

I was planning to go for RegisterFactory, but SetFactory is good for me too, it matches GetFactory indeed, I'll go for that. If the invariant name is already there, two things can be done: if the type registered with SetFactory is different, it's ignored (as there's already a factory) or it's simply always overwritten. The latter is simpler I think and more clean, as the former silently hides a misconfiguration that will go unnoticed.

Everything is thread safe, so reading always performs a ReaderWriterLockSlim.EnterReadLock inside a try / finally and the finally always calls ExitReadLock. Registration will use EnterWriteLock(). Reflection based reads of the datatable and manipulation are not covered by the lock but that's with everything done through reflection...

I thought about storing things internally in a ConcurrentDictionary and building the DataTable when the GetFactoryClasses() method is called, however for now decided against that as code which currently manipulates the DataTable through reflection will then not work, and as the code to build the datatable has to be there anyway, it's actually more work to store it separately :)

Now if I can get unstuck in the tests I can finish this...

[divega]

@FransBouma by in-memory implementation I mean that it is based on a dictionary that collects the key-values from the calls to SetFactory rather than entries in System.Configuration. A hybrid implementation would be what we will we have in .NET Framework and Mono once we port the SetFactory method there: it will have to take into account both what is in configuration entries and the calls to SetFactory.

I am strongly leaning towards having the last call to SetFactory always win :smile:

I am ok with either ConcurrentDictionary or DataTable with guards on registration. ConcurrentDictionary is that I had in mind, but I did think of reflection compatibility as a requirement. I guess you are one of the few people that wrote that kind of code :smile:

[divega]

The API indeed does this for 99.9%. In the .NET full version, it registers 4 factories, always: ODBC, OleDB, SQLClient and OracleClient. On .NET core, OleDb and OracleClient are absent, so I decided not to preserve these 2 rows in the datatable (they have predefined indexes in .net full, why I have no idea) as it would leave 2 empty rows in the table when GetFactoryClasses() is called.

@FransBouma this is another area in which I have concerns. In .NET Core in general we have been trying to follow a pay-per-play principle. It is not great if using these APIs causes multiple assemblies to be loaded in memory without knowing that the application is going to need them. I wonder if we can have the list empty by default and require the call to SetFactory even for the providers that ship with .NET Core.

[FransBouma]

@divega

The implementation I'm having now is using a datatable internally with a lock, similar to the one in netfx, with the difference that it simply creates an empty datatable at startup (static initializer of the field) instead of reading config info, as that's not there.

SetFactory simply alters that table so that should work on .net full as well (same lock code has to be added there of course).

DataTable is indeed a better choice in this to keep compatibility for code high.

I now have this API:

    public static partial class DbProviderFactories
    {
        public static DbProviderFactory GetFactory(string providerInvariantName) { throw null; }
        public static DbProviderFactory GetFactory(DataRow providerRow) { throw null; }
        public static DbProviderFactory GetFactory(DbConnection connection) { throw null; }
        public static void SetFactory<TFactory>(string name="", string providerInvariantName="", string description = "") where TFactory : DbProviderFactory { throw null; }
        public static void SetFactory(DbConnection connection, string name="", string providerInvariantName = "", string description = "") { throw null; }
        public static DataTable GetFactoryClasses() { throw null; }
    }

So the default call for SetFactory is e.g.

DbProviderFactories.SetFactory<SqlClientFactory>();

which fills in the blanks.
Or:

DbProviderFactories.SetFactory(new SqlConnection());

which does the same. You can register things under names you want as well, similar to what's there now in machine/app.config in netfx.

Having the list empty at startup is indeed simple and would prevent loading e.g. Odbc's assembly into memory for grabbing the type info / assembly qualified name, or we have to hardcode that like it's currently done in netfx (with hardcoded strings)

Having it empty does require everyone who wants to use SqlClient to register the factory at startup though. This is currently in netfx not needed.

[FransBouma]

Calling SetFactory with an existing invariantname only overwrites the AssemblyQualifiedName value, so the factory type, the rest (name/description) is kept as-is as it doesn't really makes sense to alter these at runtime and keep the invariant name, IMHO.

[FransBouma]

Please label this to 2.1.0. Code is written, api has to be reviewed, and tests have to be added.

[karelz]

Done, @saurabh500 please let me know if you disagree.

[divega]

Cc @ajcvickers

[saurabh500]

This is fine for 2.1.0

One of the aspects that we need to keep in mind is how the additional API on DbProviderFactories will work, when the application built against .Net core is run on target framework Netfx which doesn't have the additional API.

When an application target .Net core, it will find the implementation of SetFactory and will be able to function fine. However when the same application is made to target .Net Framework, the type forwarders will forward the DbProviderFactories to System.Data on Framework and there is no SetFactory there.

We can port the SetFactory method to .Net Framework, but for the releases where SetFactory is not available, there would be a runtime error.

I recommend that we have an online discussion about this addition with the API review group.

I had faced similar issues while creating an alternative to GetSchemaTable in .Net Core 1.0 where we couldn't modify the existing type. The alternatives were to create a new type altogether or come up with something of a syntactic sugar using interfaces, to expose the equivalent of the GetSchemaTable capability.

@terrajobst How do you recommend we take this forward? It would be great to have more folks looking at this proposal.

[karelz]

This issue is "port". It should not introduce new APIs which don't exist on .NET 4.6.1.

If we need new APIs, we need another issue, which follows API process (see the example there)

[FransBouma]

@karelz
We can't just port it over, as the netfx version initializes itself from the machine.config / app.config file, and there's no api to add other factories to the cache it contains. So for corefx methods have to be added to add the factories to the DbProviderFactories cache, as there's no other way possible. So 'port' isn't going to work, 100%. I've ported over the core API exposed by DbProviderFactories as well as its internals so when you're using the class on corefx or on netfx to obtain the factories, that's exactly the same. However to initialize the DbProviderFactories class' contents, on corefx an extra method is needed, which I dubbed SetFactory, suggested by @divega

These corefx specific methods (1 method, 2 overloads) have to be reviewed, I agree. I don't know how to start that, so whenever there's time to start that process, that's fine by me. :)

[karelz]

By porting over - I meant API surface.

Given the info above, I would suggest to cover the ported APIs (their shapes) in this issue. They may require different implementation on .NET Core, which is fine.
For the APIs which are "new" (at least new shape/overloads), let's create a new issue to track them. We will need the API review for them.
First step will be to create API description (see sample in API process).
Makes sense?

[FransBouma]

@saurabh500

When an application target .Net core, it will find the implementation of SetFactory and will be able to function fine. However when the same application is made to target .Net Framework, the type forwarders will forward the DbProviderFactories to System.Data on Framework and there is no SetFactory there

Hmm, though isn't the app when targeting .net core, using a .net core specific method, namely SetFactory? So when targeting another framework, netfx, isn't it expected that things will break, namely at places where the app uses a .netcore specific method, which obviously isn't present in netfx, as it's not in netstandard?

@karelz: makes sense. Will look into adding that issue for api review tomorrow.

[grahamehorner]

IMO I've never liked the machine.config and would personally prefer the DependencyInjection/Registration from a plug-in folder which is defined by the appsettings.json or similar; I also believe that the DbProviderFactory needs to have an interface defined; this can then be used to dynamically resolve the list of DbProviderFactory implementations using reflection. oh and don't forget the need to extend the DbProviderFactory so it can be used to create the Dependency classes SqlDependency, OracleDependency etc ;)

[FransBouma]

An interface is useless, it already has a base class with an interface, namely DbProviderFactory, all ADO.NET providers inherit from that class. Reflection isn't needed, that's why DbProviderFactories is there: you ask it to provide you a DbProviderFactory derived instance registered under a unique name.

[saurabh500]

@FransBouma The application using netstandard is expected to work across all netstandard compliant frameworks and DbProviderFactories will be in netstandard.
The app built on netstandard will work on .Net Core, netfx and any compliant framework implementations of netstandard.
One of the ways of tackling the registration is to introduce partial facades which will be invoked while calling the additional types on NetFx.

To tackle the limitations of type forwarding, DbProviderFactories should come in as-is with no methods added to it, so that it can be type forward to netfx. Introduce a new type for registration of the Factory and for netfx introduce a partial facade which is invoked on netfx which may not do anything and allows DbProviderFactories to return what is registered via machine.config.

An example of this is DbDataReaderExtensions.Facade.cs

csproj contains <Compile Include="System\Data\Common\DbDataReaderExtensions.Facade.cs" /> https://github.com/dotnet/corefx/blob/master/src/System.Data.Common/src/System.Data.Common.csproj#L284
In this example above for Partial Facades, the code is built against netfx and all calls to the GetColumnSchema populate the ReadOnlyCollection<DbColumn> from DataTable. This mechanism offers a sort of a bridge between the diff of netstandard APIs and the netfx versions where the API is not available.

The approach I have mentioned above is a first thought approach and needs more discussions and thoughts wrt to usability. I am mentioning this as an example of how to tackle a new API problem.

[FransBouma]

@saurabh500 ah ok, I was solely talking about the situation till DbProviderFactories is part of netstandard, hence the miscommunication I think :) The new type for registration, you mean by that a static class with extension methods, and for netfx the implementation of that is simply doing nothing? I have to wrap my head around this for a bit till I understand it completely I think... :)

[jemiller0]

-1 on doing something that requires dependency injection. I'll take config file hell over a dependency on a dependency injection framework.

[jnm2]

@jemiller0 dependency injection means passing an argument. It does not mean depending on an IOC framework.

[jemiller0]

Why don't they just add support for App.config and leave it as is for compatibility? Obviously, not having that there is going to cause a lot of other compatibility problems elsewhere as well. I don't know what the BFD is about not allowing config files in .NET Core. Apparently, there is some rule against it.

[divega]

The application using netstandard is expected to work across all netstandard compliant frameworks and DbProviderFactories will be in netstandard.
The app built on netstandard will work on .Net Core, netfx and any compliant framework implementations of netstandard.

@FransBouma I talked to @saurabh500 about this today. The idea (that I tried to reflect in https://github.com/dotnet/standard/issues/356) is that the API that allows configuring a factory at runtime will initially only exist in .NET Core. Eventually it will be ported to .NET Framework and later it can be included in .NET Standard.

The saving grace of this is that applications don't actually target .NET Standard (only libraries do), so there can be code in the application that is .NET Core specific and adds the provider configuration, e.g. on application startup. Any library code that later resolves factories should work regardless of how the provider factory configuration was added.

[FransBouma]

@divega That was my idea too hence my attempt to implement it so it could be merged in 2.0.0 and my struggles to get the tests running outside of .netstandard :). I think a bit of miscommunication resulted in the confusion that's currently surrounding this topic.

If I can get some tests running on .netcore the code should be ready to go (but that's currently not working due to tooling issues)

[karelz]

@FransBouma I was under the impression that latest info should have unblocked you. Is that not the case?
If not and you are not getting any traction, please ping me on the main issue where you track the problems and I will bump it again. Thanks! And sorry for the troubles (you had rather large share of them).

[FransBouma]

@karelz to be honest I haven't tried as it was scheduled to be .netstandard 2.1 anyway so I thought it would be better to wait till 2.0.0 hits and then take another stab at it, as tooling would be mature by then (and it wouldn't be merged before that anyway ;)). Will try later today :)

[karelz]

Tooling won't mature on its own sadly :(. My latest understanding was that you were blocked on lack of documentation how to add netcoreapp-only APIs. That should have been slightly better after @tannergooding kindly added it to docs notes (the docs are likely in wrong section, because I pointed Tanner to the wrong section - it should be in the one above, but the principle is the same, there is overlap between the 2 anyway)
If that doesn't help you, I'd like to know details of what is blocking you.

Note: master branch is already opened for 2.1 work. Last integration into release/2.0.0 branch from master happened last week. We already changed master branch versions to 2.1.

[FransBouma]

@karelz I've tried that to the letter, but it still doesn't allow me to compile the tests inside vs2017 ('DbProviderFactories is unknown'), no matter what I do. It might be I need 15.3 preview installed (which I haven't and can't it's a production system I need) but no-one has been able to answer me this.

So I don't know how to write software using this. I know it's likely something small I'm overlooking, but what that is, I have no idea. I've committed the changes I've made to the PR, so others can see what I have and what might be the problem, I am out of options.

It's really simple:

1. A simple list of do this to get things to compile, including test projects. This is a simple list of what tooling and what versions are required.
2. A simple list of do this to run a single test, be it on the command line or not, I don't care
3. A simple list of do this to debug a single test. If that has to be done in vs code or x64dbg or Ida I don't care, I am used to stare at x64 assembler in a debugger, but there's no info how to debug a single test. None.
4. A simple list of do this to run all your tests so you can confirm things work.

At the moment it's, sorry, chaos. The fragments are scattered, they're there but not known to people outside MS like myself, and it's a puzzle in which order which fragments have to be read/used and which ones are outdated.

That's why I thought it would be better to wait till things are more settled down so an outsider like me, who doesn't know details of the scripted system in place to make it all compile and work, can simply put code into some files, configure things based on easy step lists and contribute without having to do the trial/error dance.

As it's at the moment no fun at all.

[karelz]

I understand your frustration, let's look into it deeper.
One thing that strikes me a bit is that we had maybe 50 new contributors contributing to CoreFX this year and this is first feedback of this type ... so I wonder (truly, without any attacks or anything), if your standards are a bit higher? ;)
Or maybe you (like me) don't read XXL-long steps and if it isn't clearly described somewhere on first page / self-explaining, then you are stuck? BTW: I do have exactly this approach myself, that's why I started the new docs effort where I am trying to optimize the common few paths to be super-simple. The truth is the docs are not ready for someone like me yet.
cc @haralabidis

[FransBouma]

@karelz no my standards are pretty low in this regard. I wrote a complete linq provider without any docs from MS on what Queryable was at the time, 9 months of full time trial/error. That's why they call it software engineering, right? As a hobby I even reverse engineer 3D game engines to add camera tools to manipulate them at runtime through dll injection and assembler/C++. I don't care if there are no docs, but at least that there is some idea what to do. Like I said before to you, I've spent almost a week full time on this, with 90% fighting with tools. I did that because I didn't want to give up easily. So no, that's definitely not it.

Here I simply don't know what to do anymore to make it work.
Is it the vs tooling? No idea, no-one told me or can tell me, I've asked several people, even ones on the csproj team. I can try of course, but with the risk of ruining my dev box I need to deliver software with for my customers and wasting time afterwards because it's a preview and with all things preview from Microsoft, removing them is likely going to be a pain.
Is it some configuration setting? Again, no idea. You pointed me to the documentation fragment which describes csproj settings I haven't seen before. It didn't make it work, but it's progress, I guess.
Right now, I can't get the test to compile as it references the wrong BCL. Why that is is anyone's guess.

I did read every doc I could find on this, as I didn't want to come across as this arrogant dev who thinks he knows best and runs in without RTFM. But here I am, frustrated after spending all this time on this, with no way to compile a simple test, left alone debug it. (Yes, I learned from an issue in the xunit add-in repo for vscode of all places that one could debug a test by adding a while(!Debugger.Attached()) to the test to make you debug it, I even tried vscode but the xunit add in can't (yet) deal with conditional includes in the csproj.)

One thing that strikes me a bit is that we had maybe 50 new contributors contributing to CoreFX this year and this is first feedback of this type

I must say this didn't get down well with me, at all. How do I have to read this other than that it looks like I'm some pampered vs oriented dev who drags/n/drops his code together?

[karelz]

@FransBouma I didn't try to question your technical capabilities, experience or willingness to push through. I fully trust that you are super-b engineer, you proved that already before. I was honestly trying to find what is different on your approach than what others went through. It doesn't speak anything about you, it speaks about what tools you used and how and that somehow that went different path than others. I totally expect that all this is just some gap in our engineering experience, I just don't know where yet.
I'll work with you offline on figuring out what exactly are your steps and how they differ from others to find out where we have gaps. Maybe this is specific to SqlClient. Maybe it is specific to VS usage. Once we find out, we will address it, because I bet there will be many others who will fall into the same traps as you did here.

I definitely appreciate your time and patience so far, you invested more than 1 week full time on making it work. It is way above anything I would expect from you or anyone else to do. And believe me, we appreciate your effort. It won't be total waste of time as it will help us shape docs for future contributors.

[jnm2]

@karelz

One thing that strikes me a bit is that we had maybe 50 new contributors contributing to CoreFX this year and this is first feedback of this type ... so I wonder (truly, without any attacks or anything), if your standards are a bit higher? ;)

No, I agree with @FransBouma. The practices that have become obvious to regular contributors are still really confusing and time consuming to grep, even when you move to doing different kinds of contribution like API addition. Running and debugging tests is tedious and I couldn't always get it to work. Just a data point.

Every time I have asked for some straightforward docs on each type of procedure, I hear that your team practices are evolving too fast and people don't have time to document it. (And then I am welcomed to document your practices, even though I'm never going to have the full picture even when I figure out everything necessary for my contribution.)

It is essential to maintain and evolve crystal clear and comprehensive step-by-step contribution instructions along with the actual projects and build scripts. Having that makes your project more attractive to outside contributors and is more considerate of their time, to put it bluntly. I've contributed to a number of projects and this is the single biggest thing you notice about each project.

Sure, I'll contribute again because are you kidding, this is the BCL! =) It's worth the effort.

[karelz]

Thanks for feedback @jnm2! It helps to know others are/were facing similar problems. It will help us bump priority of the docs.

Good new contrib docs is something I have on my todo list for quite a while. I hope that I will be able to ask someone on the team to help improve them during June.
So I personally understand you and can't agree more. The more contributors who confirm the feedback from you & @FransBouma, the easier it will be for me to get funding ;).

I believe we passed the point of "we are evolving too fast to have docs" in January. Our infra is pretty decent now and I don't think we can afford that excuse anymore. VS integration for contributors is lacking, we know that. At minimum we should call out huge warning in the docs. Ideally we would fix VS and then react to every single report of things not working.

cc @weshaggard @ericstj @mellinoe @danmosemsft

[jnm2]

@karelz Thanks for considering! I saw a similar comment here: https://github.com/dotnet/corefx/issues/20325#issuecomment-305127003

[karelz]

@jnm2 that feedback is about using .NET Core 2.0 to develop libraries. It is different than the topic here - contributions to the CoreFX repo itself.
While they seem similar, there is huge difference - CoreFX has its own build system, test runner, etc. We do not reuse the same things that .NET Core 2.0 developers use. The requirements to build the platform (CoreFX) itself are slightly different from building on top of the platform (libraries targeting .NET Standard / .NET Core). The timeline is also different - the platform itself needs the support from day 0 to build the platform itself, while building on top of the platform can often wait for later, when the platform somewhat exists.

[jnm2]

Okay, my bad. I thought ImageSharp was a corefx-like project.

[willdean]

As one of those 50 new corefx contributors, I don't think I can allow myself to be co-opted in wholehearted support of the way things are.

I found corefx very difficult to work on:

• I opened/contributed to a number of issues about tooling and doc issues faced by the newcomer
• I politely discussed ways to work around xUnit's miserable assertion API
• I politely discussed ways to work around xUnit's pathetic trace collection problems
• Like all good subjects, I refrained from pointing out that xUnit has no clothes.
• In order to actually debug tests I created new projects from scratch in VS, link-inserted all the relevant bits of corefx into them and then debugged in there. Only at the end did I go back to the execrable command line test runner experience.
• I maintain a text file of preposterous command lines which are needed to run various bits of the test framework in various ways.

For balance, I'll say that my very positive experiences of working on corefx stem from the friendly and helpful attitude of the corefx MS guys on GitHub. I wish I could have @karelz to do first-line support on every OSS project I've worked on, and there were lots of others too...

I actually do get it why the framework's own development couldn't start from exactly the same tooling position as end-users will ultimately use, and corefx probably feels the need to outpace the lumbering old donkey which is VS. But corefx is basically only building class libraries, so I don't see why it needs to be so far in left field. However, I can't help thinking that if the corefx team actually had to develop corefx code in the same way as every other .NET library is developed (i.e. in VS, with 3rd-party test runners), then both VS and .NET Core would be better for it.

I know this is off-topic here but, genuine question, where would it be on-topic? Can I actually usefully open an issue in corefx's GitHub repo saying (more politely) "The VS .NET Core dev experience is unutterably awful"? The scale of the problem is way beyond filing VS bugs, even if I did fancy exchanging platitudinous snippets with the staff of a Chinese outsourcing company.

@jnm2 Has a perfectly valid point - these is absolutely no good reason at this time why the 3rd-party development of a library like ImageSharp should look any different to the MS development of the vast majority of corefx.

I'm ready for my telling-off about unprofessionalism now...

[FransBouma]

@willdean

In order to actually debug tests I created new projects from scratch in VS, link-inserted all the relevant bits of corefx into them and then debugged in there. Only at the end did I go back to the execrable command line test runner experience.

I think I lack this knowledge at the moment. Is there a page you can point me to so I can gain that knowledge and create these projects myself too? I know I can look at other project csprojs, and have done so, but as the machinery uses .props files I don't know how to do this properly. One question I haven't seen answered and which is related is: you do need 15.3 update for vs2017 for this? Thanks

I maintain a text file of preposterous command lines which are needed to run various bits of the test framework in various ways.

Would you like to share those for others to use? That would be great :)

I think it's a matter of once you 'know what to do and have the info bits to navigate around the dark pits of hell' it's easy to contribute and you forget you ever had problems. To get there is a bit of a struggle ;)

For balance, I'll say that my very positive experiences of working on corefx stem from the friendly and helpful attitude of the corefx MS guys on GitHub. I wish I could have @karelz to do first-line support on every OSS project I've worked on, and there were lots of others too...

Seconded

I also agree with the assertions you made regarding xunit and it's... let's say 'less ideal ways of doing things' to stay polite. I've never seen a framework that has one job and do it in such an user unfriendly manner. Almost as equally unpleasant as vi on an SCO unix prompt. Oh... did I just slip off the 'politically correct' path there? ;)

[willdean]

@FransBouma I don't think I have anything useful to share really - the stuff I worked on was a direct port of netfx code, and so to debug the tests I built them as a netfx project against the original netfx implementation, debugged the tests in netfx land and then went back to corefx.

I also never had to create a corefx project from scratch because someone in MS did that, so I can't offer any advice there, other than that I'd probably just copy a simple existing sub-project within corefx and run around changing names, etc.

I was using VS2015 at the time with corefx - I wasn't clear about VS2017 support in corefx at that point (earlier this year). I would be very wary of installing the 15.3 preview - it's billed as a safe 'side-by-side' install, but on my machine it trashed all the VS2017 15.2 settings back to defaults which is a huge pain, and the 15.3 preview crashes every 30 seconds anyway, so I didn't find it useful for anything.

Here's one of my test-running command lines, though I doubt that's really where you're stuck, and if you're on a development box then there are easier commands to run all the tests in a project which are part of the corefx build system. (Still all command-line though).

N:\will\corefx\bin/testhost/netcoreapp-Windows_NT-Debug-x64\dotnet.exe xunit.console.netcore.exe System.IO.Ports.Tests.dll  -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapp1.1tests  -parallel none -showprogress

If you can get your tooling expectations dialed-back about 25 years (but without the 8.3 filename restriction) then you might find less disappointment.

[FransBouma]

@willdean Thanks, any info is good info :) Especially regarding 2015 and 2017 15.3. It tells me the stuff I have in place should be sufficient, and that's not it. If no-one from MS has time to solve it, I'll indeed try to copy an existing project and see whether I can get that running.

If you can get your tooling expectations dialed-back about 25 years (but without the 8.3 filename restriction) then you might find less disappointment

heh :) good point. I'll dial it back to the experience with what I used back then haha ;) (vi editing, typing cc or make on command line for compilation, for debugging gdb without source overview, on a 80x24 WYSE terminal)

[jnm2]

For the most part it was fine but there were aspects of using xUnit that I didn't enjoy, also. I don't expect that they're interested in switching but as a data point, while I was contributing I wished I was using NUnit so many times. (This was before I became a member of NUnit, so no disclaimer needed.)

[karelz]

First, thanks everyone for the feedback - I am ABSOLUTELY interested in this kind of feedback! Now, let's do something about it ...

I suggest to move the discussion into separate issue - that is totally appropriate way to discuss "how to contribute to CoreFX". I would suggest to split the discussion into 2 parts - one for contributor experience hassle (incl. VS) (see dotnet/corefx#20570) and second about xUnit (TODO anyone?).
Motivation: You might have guessed that changing fundamental part of our infra as xUnit, is a LOT of work and while I am interested in feedback there as well, it won't change overnight and potentially never (due to high cost) -- either way, it is good and healthy to have that discussion anyway.

[danmosemsft]

I suggest in those issues we try to get wider consensus on a prioritized list of improvements. There are lower hanging fruit than xunit, I think.

[willdean]

I don't think there would be any need to replace xUnit, just have some of the more painful omissions fixed - I think there is some (historical?) connection between xUnit and MS, so there may be some encouragement could be brought to bear there?

[karelz]

@willdean if there are painful omissions in xUnit, please file a new issue, so that we can understand them and then act on them. Thanks!

[ghost]

Nice to see I'm not alone in my xunit non-fandom. Here are my issues (from a very cmd-line bare-metal developer when it comes to tests.)

https://github.com/dotnet/buildtools/issues/1538

Looking forward to that xunit thread...