Runtime: Port DatasetExtensions to .NET Core

[hugosmarques]

Anyones knows if .AsEnumerable() propertie will be available for datatable on .net core

[danmosemsft]

This is a request for DataTableExtensions.AsEnumerable which is in System.Data.DataSetExtensions.dll.

@saurabh500 is this available for Core? I don't see it in our sources or NuGet.

[saurabh500]

@danmosemsft DataSetExtensions was not ported to Core.

[danmosemsft]

Is it something worth considering - would it be a fit for CoreFX? We have had success with dumping desktop sources and community folk driving the work.

This is the only ask I have seen for it though.

[hugosmarques]

@saurabh500 did you know if it will be available in the future? I have to know because if this is not available I have to abandon the .net approach.

[saurabh500]

The source for DataSetExtensions is at https://github.com/Microsoft/referencesource/tree/master/System.Data.DataSetExtensions

We haven't planned porting this particular assembly to .Net Core. However I do agree with @danmosemsft idea about having the community bring the code to .Net Core. I don't see any platform specific dependencies in the code so this should be relatively simple.

[danmosemsft]

Would we be open @saurabh500 to having it in CoreFX? If so, we can invite a PR to get things started.

[hugosmarques]

Well guys I do not have words for your good vibe in helping me. I will be very thankful if this is implemented, because right now for me is impossible to Port all my code to EF, I have thousands of line code and I use datatable with Linq. If is simple as @saurabh500 says I'm praying that it's implemented.

[danmosemsft]

@karelz fyi another port-to-core request.

[karelz]

Moving to Future.

Next step: If there are folks interested, we can dump the code and let community make the code build, add tests, then ship it. Marking as up-for-grabs.

[hugosmarques]

If the integration of this is relatively simple i hope that someone do this ASAP please.

[hugosmarques]

is not supose that if .net core have DataTables properties gave the AsEnumerable propertie too, otherwise how i can use LINQ with datatables

[karelz]

@saurabh500 @NickCraver @FransBouma do you think this is must-have for 2.0? Or can it wait for Future?

[FransBouma]

I use it with meta-data filled DataTable instances (obtained from e.g. GetSchemaTable, stored procedures or ad-hoc sql queries) so I can write Linq queries consuming the meta-data. I use this method as it was available on in .NET full (the code is in .NET full targeting drivers), however I think datatable.Rows.Cast<DataRow>() would do just as well as .AsEnumerable(). EnumerableRowCollection, the class handling the enumeration of AsEnumerable(), has more logic than just a yield return or a cast of Rows, but in AsEnumerable() none of that is used, so it basically comes down to datatable.Rows.Cast() (as far as I can see by reading decompiled code, not tested with running code so I could be mistaken!)

So TL;DR: no, not a must have, as the workaround is .Rows.Cast(). I think the group of people targeting a DataTable with Linq is pretty small (ORM devs, using metadata in datatables with linq), and therefore there's likely not a lot of people hurt by this missing method, but as it's a simple .Rows.Cast(), it might not be a burden to port the method over as that instead of the whole EnumerableRowCollection class.

[hugosmarques]

@FransBouma tell me one thing. I see that i can use:

datatable.Select().Select(x => ... instead of datatable.AsEnumerable().Select(x => x.....

Exist any diference between the two cases?

[karelz]

Thanks for advice @FransBouma!

[FransBouma]

I see that datatable.Select() creates a new array and datatable.AsEnumerable() wraps the rows in the datatable, similar to datatable.Rows.Cast<DataRow>(), so the former copies data, the latter wraps it, which can have a huge impact.

[hugosmarques]

@FransBouma so please if as you said is a very simple feature can we have this for the 2.0 version?

[FransBouma]

@FransBouma https://github.com/fransbouma so please if as you said is a very simple feature can we
have this for the 2.0 version?

I don't work for Microsoft, so I can't decide over that :)

[karelz]

@hugosmarques unless it is super-critical for many customers, we will not add more APIs into .NET Core 2.0, sorry. See details in https://github.com/dotnet/corefx/issues/17619#issuecomment-301937346

[hugosmarques]

@karelz thanks for all but i cannot understand this decision.... as we all know this is a simple integration and when .net core team decide to put the DataTable on .Net Core i did not understand why does not put the AsEnumerable propertie. People that use DataTable needs this. This is a disapoiting final for me because i cannot move forward with my migration to .Net Core without this.....

[hugosmarques]

@FransBouma if i use dt.Rows.Cast().Where(a => instead of dt.AsEnumerable().Where(a =>
is the same or almost the same right?

I make a test with 1000000 rows on datatable and the time for both are the same. Did you see any incovenient of using dt.Rows.Cast() instead of dt.AsEnumerable()

[FransBouma]

No, I think they're the same, functionality wise, as far as I can see.

[hugosmarques]

@FransBouma this is what I need from beginning... with this I don't need to port AsEnumerable to .net core
dt.Rows.Cast() is available on .net core 2 right?

[FransBouma]

Yes. Just add using System.Linq; at the top of your own code file to make Cast available.

[hugosmarques]

@karelz thanks again. We have a solution for this:)

[danmosemsft]

@saurabh500 @corivera @weshaggard any objection to me finding someone to just go ahead and do this now?

Also I assume corefx is a good place for it, given it's small and has no home right now - also closely goes with S.Data.

@terrajobst

[saurabh500]

I don't have any problems. Are you gonna own porting it Dan?

[danmosemsft]

If it's simple we can port it. You guys would own it after that (which doesn't sound like much of a burden)..

[nchikanov]

Port of System.Data.DataSetExtensions completed! (#22669 port, dotnet/corefx#22892 tests, dotnet/corefx#23087 pkgproj)

[danmosemsft]

@hugosmarques this is not on nuget yet, @nchikanov will look into that...

[joelbraun]

@danmosemsft I have a porting project I'm very excited to use this library on. Any ETA on getting this on Nuget? Thanks!

[danmosemsft]

I believe we are waiting on @weshaggard to fix the versioning in master first.

Meantime, can you make progress using myget -- it may work to just add myget to your nuget config, see dogfooding. Or since they are simple extension methods, might be easier to just temporarily add the C# files to your own project.

[manhao-chen]

so to clarify as i'm new to core coming from framework

is this gonna be available/released as an update of .net core/standard or could we get it as a package from nuget?

[danmosemsft]

@manhao-chen this will likely be on nuget in stable form later in the year. In the meantime, I recommend you just add the source files to your project -- they're extension methods so this iwll work fine. When the package is out, you can ref that and remove the sources from your project.

[manhao-chen]

@danmosemsft could you point me to the extension cs files please

[danmosemsft]

https://github.com/dotnet/corefx/tree/master/src/System.Data.DataSetExtensions/src

[kfarris9]

@danmosemsft I've been tracking this post and I'm a little confused. I see the src for DataSetExtensions on github, but your post implies it isn't available until later in the year. I'm using a recent nightly build of 2.1, but are you saying I still need to copy the src to my own project and use it from there until some future data? I'm a little shy about copying src into my local repo for our product and making the rest of my dev team think thats something we support going forward.

[eriawan]

@danmosemsft

I believe we are waiting on @weshaggard to fix the versioning in master first.

Pardon, how is the progress on this versioning fix?

Since it's been more than 3 months without any update, are there any plans or timeframe for System.Data.DatasetExtensions to go into .NET Core 2.1 Preview2 package? Even after I try to download the latest package of .NET Core 2.1 Preview 2 from MyGet, it still has no System.Data.DatasetExtensions package available. CMIIW, though...

[weshaggard]

Please ignore the preview2 packages on our myget feed. You can get the latest changes under preview1, see https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Data.DataSetExtensions/4.5.0-preview1-25927-02.

[eriawan]

@weshaggard
Thanks for the reply! 👍
Any scheduled ETA for this System.Data.DatasetExtensions package? Will this package be included as part of .NET Standard 2.1/.NET Core 2.1? Or will it be available as standalone nuget package?

[karelz]

There is no .NET Standard 2.1 planned. Just .NET Core 2.1.

This package is part of Windows Compatibility Pack for .NET Core.
We do no have official schedule yet. I expect it will ship at .NET Core 2.1 time frame the latest - see .NET Core 2.1 high-level roadmap.

[eriawan]

@karelz
Thanks! Out of curiosity, any reason why is this package included in Windows Compatibility Pack? I have used it from myget, and afaik it doesn't have any dependency to Windows platform at all, therefore it should be good/OK to be included into .NET Core.

[karelz]

In general, we are trying to stop shipping individual packages where we can - they are very high cost for us compared to the benefit they bring. (e.g. test matrix explodes exponentially)
Shipping in standalone Windows Compatibility Pack allows us to provide value also on .NET Core 2.0, without forcing people to upgrade to .NET Core 2.1 right away.

[joetherod]

The preview package does not have support for AsDataView() for EnumerableRowCollection.
When will this package be complete?
@weshaggard

[weshaggard]

@joetherod it doesn't look like this library has the AsDataView APIs at all. @danmosemsft any ideas on why those APIs are missing? It was mentioned in the PR to add this library https://github.com/dotnet/corefx/pull/22669 but with no details on why.

[danmosemsft]

I tried adding it just now and it looks like it was because it ends up needing internals of System.Data.Common (eg internal interface IFilter)
In NETFX S.D.dll has [assembly: InternalsVisibleTo("System.Data.DataSetExtensions, PublicKey=00000000000000000400000000000000")]

It looks like we only use assembly IVT for test purposes. Thoughts?

[danmosemsft]

I can confirm it's only these missing:
c# public static DataView AsDataView(this DataTable table); public static DataView AsDataView<T>(this EnumerableRowCollection<T> source) where T : DataRow;

[JeremyKuhne]

I tried adding it just now and it looks like it was because it ends up needing internals of System.Data.Common (eg internal interface IFilter)

That was the reason

[joetherod]

Looking at the 4.7 code, it may need this a well?
internal class LinqDataView
Timeframe when added?
And when I try the latest nupkg:
https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Data.DataSetExtensions/4.5.0-preview2-26227-01
It does not install. It errors out.
I can install the one available in the Nuget repo marked as prerelease.

[danmosemsft]

@joetherod yes LinqDataView is part of the implementation.

We don't have a time frame because it's not planned. I would like to add it somehow. Perhaps this assembly could merge into S.D.Common.dll itself -- it's quite small. Then it could be added. @weshaggard ?

[weshaggard]

Perhaps this assembly could merge into S.D.Common.dll itself -- it's quite small. Then it could be added. @weshaggard ?

I don't know enough about the details but it is certainly something we could consider. If we do however it will become part of the shared framework and not an OOB package, that may be OK but something to call out.

[danmosemsft]

@joetherod I recall when we cut this, we looked at telemetry and the use of AsDataView was really low. Can you avoid using it for now, is that feasible? If not you could possibly paste the code into your app (consistent with license) and use reflection for where it reaches into System.Data.Common internal types.

[joetherod]

The app Im working is a port from MVC 4. Its a tenant with over 400 public facing sites. It has 1000's of razor views that are in the database. And this code is unfortunately used in some of the razor code. I've been creating extensions, sub classes, and trying to support all of the missing components in core to get this working. Eg. Response, Request, Cookies, etc.
I tried porting over the extensions but got stuck on the this class, LinqDataView . Seems to be needed as well to do the AsDateView() method. If you can point me to the write code files to get this working, I will try again to port it over.
Thanks

[danmosemsft]

Code can be found in reference source which can be most easily browsed here

But, I looked a little further, and it would be difficult, even with reflection. LinqDataView overrides DataView in several places, and calls into base implementations, which themselves access private members... and you cannot paste in DataView itself, as it is returned by the AsDataView methods, and it would be a distinct type to the "real" dataview.

Can you dig deeper into what it would take to snip the dependency on LinqDataView?

[joetherod]

The reference source doesnt seem to have the latest extensions code. The other source, I had trouble getting the LingDataView class to get all it needs. I think it was probably because it was 4.7 and my system,data is a different version. Are there links to the previous versions of the framework source code or do I have to download the whole thing?

[danmosemsft]

@joetherod which code do you believe is out of date? I just diffed the latest internal .NET Framework sources with https://github.com/Microsoft/referencesource.git and the relevant sources seem to match

[joetherod]

This file, https://github.com/Microsoft/referencesource/tree/master/System.Data.DataSetExtensions
Where are the extension classes?
Is there a chance you port this over in the preview release of System.Data.DataSetExtensions.
And fix the nuget packages?

[savornicesei]

@joetherod You can find the package here along with instructions on setting up the myget feed in your environment.
I have added the System.Data.DataSetExtensions project (from this repo) into one of my projects and it complained about DataSetExtensions class so I removed it. I also replaced the whole thing with the NuGet preview package and it worked fine.

All the best,
Simo

[joetherod]

@savornicesei I was able to install the package, but it still doesnt have support for AsDataView for EnumerableRowCollection

[danmosemsft]

@joetherod which specific type or file is missing for you? eg
https://github.com/Microsoft/referencesource/blob/90b323fe52bec428fe4bd5f007e9ead6b265d553/System.Data.DataSetExtensions/System/Data/LinqDataView.cs

Is there a chance you port this over in the preview release of System.Data.DataSetExtensions.

If we merge the assemblies it is possible, but priority wise you are the only person who has asked for these to date as far as I know. Is anyone else blocked on them?

[joetherod]

LinqDataView.cs and the extension methods for AsDataView() in DataTableExtensions.
When bringing in the LinqDataView.cs, the base class isnt in sync with my version of Dataview.cs in core.

[danmosemsft]

I linked to LinqDataView.cs above, AsDataView is here:
https://github.com/Microsoft/referencesource/blob/90b323fe52bec428fe4bd5f007e9ead6b265d553/System.Data.DataSetExtensions/System/Data/DataTableExtensions.cs#L258

I checked these sources for DataSetExtensions exactly match the .NET 4.7.1 sources. .NET Core sources are often different at least for formatting.

[joetherod]

The code in core doesnt exist for dataextensions, so not sure what you mean. I copied over all of the classes from 4.6 and 4.7. The LinqDataView.cs has errors with it base class ctor implementation which is DataView.cs.

internal LinqDataView(
DataTable table,
Func predicate_func,
Predicate predicate_system,
Comparison comparison,
Func comparerKeyRow,
SortExpressionBuilder sortExpressionBuilder)

            //Parent constructor
        : **base(table,
            predicate_system,
            comparison,
            DataViewRowState.CurrentRows)**
    {
        this.sortExpressionBuilder = (sortExpressionBuilder == null) ? this.sortExpressionBuilder : sortExpressionBuilder;    
        this.comparerKeyRow = comparerKeyRow;
    }

[danmosemsft]

The code for DataSetExtensions is in .NET Core -- it's at https://github.com/dotnet/corefx/blob/master/src/System.Data.DataSetExtensions/src

If you copy over the code from reference source, including LinqDataView, you will hit the issue I mentioned that it requires access to internal members of the base class, and the base class is in System.Data.Common.dll. In .NET framework this was OK, because the assembly of the base class (named System.DAta.dll there) exposes its internal members to System.Data.DAtasetExtensions.dll.

Does that help? I apologize because I feel like I'm not udnerstanding well.

[joetherod]

@danmosemsft I understand now. So, this cant be done by me. Can someone take care of this in the prerelease package of DataSetExtensions then?
Thanks

[danmosemsft]

@joetherod since it requires merging the assemblies it's not trivial work and will probably not happen for 2.1 given the low usage of this API unless it emerges that a bunch of other customers are hitting it also. Hopefully you can find a workaround.

[danmosemsft]

Since this issue is closed I opened https://github.com/dotnet/corefx/issues/27610 to track it. Lte's continue any conversation there.

[divega]

Since this issue is closed I opened dotnet/corefx#27610 to track it. Lte's continue any conversation there.

@danmosemsft actually this issue doesn't appear to be closed. Was the intention to close it?

[ToddBradley]

@danmosemsft Low usage? Every single Stack Overflow explanation of how to create a subset DataTable from an existing one relies on AsEnumerable, which doesn't exist in .NET Core. The only workaround seems to be rolling your own subsetting code in a big for loop, which is ugly and messy.

[OHAVM]

Since this issue is closed I opened dotnet/corefx#27610 to track it. Lte's continue any conversation there.

With respect, this is going to be a pain point for me and it appears that the other issue was closed with a "this is too hard and we're lacking resources. Sorry". I will say though, @danmosemsft , if your plan is to not implement this causing .NET Framework developers converting to .NET Core to code around the lack of Data Table extensions, causing the need for DataTableExtensions to drop off meaning that your team won't have to actually address this issue, then I have to respect your logic even if I'm at the unfortunate end of it.

[danmosemsft]

@OHAVM the only reason this was not done is convoluted dependency problems. You can see some discussion in dotnet/corefx#27610 . I have driven many ports of NET Framework features and it is pretty rare that we encounter such problems.

We don't have plans right now to have another run at the problem but I reopen this so we can continue to get feedback. As well as leaving a comment I encourage folks interested in this to do a thumbs up of the top comment. That's a datapoint we've used in the past.

[MSiccDev]

We are currently running into issues with the AsDataView() extension method as well. One of our private third-party NuGet packages needs this method internally and is, of course, throwing MethodNotFoundException there.

Microsoft wants us to port our code base over to newer versions of .NET, but puts not only stones, but rocks in our way. In my opinion the low number of complaints is just because others have thrown the towel on porting their code base already earlier, so they never reached here. Just because you do not see a lot of telemetry data regarding to certain code parts, it does not mean they are not essential. There are effective methods to block telemetry data submission, which are often used in enterprise applications.

[bmperdue]

Really need the AsDataView() Added this is causing a lot of issues getting our code to work.

[don4of4]

@danmosemsft Based on a cursory review, it looks like this relatively simple feature is causing a lot of pain.

Could you update us on the status of this? _The ticket you referenced is a dead link._

[danmosemsft]

Fixed link - it was https://github.com/dotnet/corefx/issues/27610. There are more details in the older PR https://github.com/dotnet/corefx/issues/31764.

[ericstj]

Fixed in dotnet/corefx#36528

[danmosemsft]

@don4of4 @hugosmarques @OHAVM @bmperdue and all other folks above looking for this package, I'd be grateful if you'd try it out in our Preview 4 release (not too far away) so that we can be sure it solves your problems (and if not , we can fix it)

Thanks @ericstj for porting.