Standard: Idea/Proposal: Split .NET Standard into multiple smaller standards by relying on tooling

[GoldenCrystal]

First, let me start by saying that this is not supposed to be a final proposal, but more like a rough draft / general idea, that I hope to tranform into a valid proposal with the help of everyone here.
I fear this is going to be a controversial topic, but I've had this idea in the back of my head for a few days now, and I'd like to at least try explaining it there, so as to have your feedback.
So, please bear with me until the end of this topic. 😅

Abstract

Like many others, I do wish .NET Standard 2.1 was already a thing.
But I do also perfectly understand the concerns about .NET Core 2.1 being the only implementation already ahead in the standard. And because of that I do fear that .NET Standard 2.0 will be the last .NET Standard version ever. (But at the same time, I'm kinda going to propose replacing it by something else here… 😒)

Aside of that, the fact that .NET Standard 2.0 is not compatible with .NET 4.5 can sometimes be an obstacle in the migration process to .NET Core. (That's something we are faced with everyday in my current job)

So I wondered, what if, instead of having one unique standard, there were many smaller standards that could be implemented à-la-carte by runtimes/frameworks/NuGet ?
What if we could already define something such as a "Span Standard 1.0" ?

And yes, I know that the original design of DNX / .NET Standard 1.0 was more modular, I know that there were once such a thing as Assembly Neutral Interfaces, and I know all of this did not work out that well. This did obviously lead to the acceptable compromise that is .NET Standard 2.0.

But, I assure you, I am not proposing to reiterate the exact same mistakes as in .NET Standard 1 era. (Which is not to say that I'm not taking strong inspiration on what has/had already been done. 😉)

What I'm thinking of is a mechanism that should help reduce coupling without requiring changes in the existing runtimes, and could work reasonably well with support in the tooling.

The general idea

We would define many feature sets that can overlap eachother. At the begining, features would likely be defined based on current .NET Standard 1.0~2.0 APIs, by splitting unrelated features away from eachother. (Basically what I call a feature here is like a much smaller .NET Standard)

A feature would be defined very similarly to a reference assembly (i.e. no implementation), and define what API a consumer should expect when depending on the feature.

I'll take as an example the current hot topic of Span<T> and .NET Standard 2.0/2.1 to illustrate how features could help:
Span<T> is available to altmost anyone, but only .NET Core 2.1 supports the fast Span<T> and the new Span-based APIs:

• Assume that there is no such thing as .NET Standard
• We could define Feature.Span, Version=1.0 for the portable span, as implemented in System.Memory
• We could define Feature.Span, Version=1.1 for the fast span, as implemented in .NET Core 2.1
• Let's assume that we already have Feature.Sockets, Version=2.0 representing the System.Net.Sockets feature from .NET Framework 2.0.
• We could define Feature.Span.Sockets, Version=1.0:
  
  
  
  • This feature would have a feature dependecy on both Feature.Span, Version=1.0 and Feature.Net.Sockets, Version=2.0
  
  
  • This feature would define all the new Span-related members introduced on Sockets
  
  
• Let's assume that we have Feature.IO, Version=4.5 representing the System.IO subset from the NuGet package with the same name.
• We could define Feature.Span.Streams, Version=1.0:
  
  
  
  • This feature would have a feature dependecy on both Feature.Span, Version=1.0 and Feature.IO, Version=4.5
  
  
  • This feature would define all the new Span-related members introduced on System.IO.Stream
  
  
• We could go on with all the other places where Span-related API were introduced, but I'm sure you get the idea…

ℹ️ Note: Don't pay too much attention about the fictional boundaries of the features yet, I'm pretty sure I wouldn't want the feature to be shaped _exactly_ like described above. 😉

With such features defined, I could write a database client library, and declare that it depends on Feature.Runtime, Feature.CSharp, Version=7.3, Feature.Net.Sockets, Feature.Span.Sockets, Feature.Collections, and Feature.Data.Common.
This would allow my library to run on any runtime, provided that it at least supports all the features I used.

The (Draft) Proposal

Defining features

A feature is an API shape, or contract (similar to .NET Standard), whose concrete implementation is provided by either a runtime (e.g. .NET Core) or by NuGet packages.

We would ideally reserve a prefix on NuGet for official .NET features. (e.g. Feature.)

• In the initial version, features would only ever be created by Microsoft / .NET Foundation
• A feature is represented as a single assembly, typically exposed as a NuGet package.
• A feature assembly contains only metadata (no implementation), like would any reference assembly.
• A feature assembly contains only public types and members. (public types, public members and protected members)
• A feature assembly can (will) have dependencies on other feature assemblies. (Dependencies would be supported via NuGet packages)
• A feature assembly defines only the exact API surface that it supports
  
  
  
  • A feature can only provide complete interface definitions. (Interface versioning problem: it is not possible to add or remove interface members)
  
  
  • A feature can provide delegates. (Complete definition, not that any other form would be valid anyway)
  
  
  • A feature provides only partial type definitions for structs, classes, and enums: Only publicly visible members that are provided by the feature, are included in the metadata.
  
  
  • features don't need to provide public constructors to type they augment
  
  
  • A feature cannot add an abstract method to a pre-existing type (TBD: How can this be enforced ? Maybe based on the presence of a public constructor ?)
  
  
  • (Obvious) Every type that is referenced in the feature assembly must exist either in the feature assembly itself or in one of its dependent feature assemblies.
  
  
• Multiple features can provide the same member on the same type, if they both provide it for different reasons
• Across feature assemblies, type identity is only determined by their full name
• features follow Semantic Versioning (TBD)
  
  
  
  • A feature of minor version M > N must always depend on the feature version N (Version M should only include the new types and members)
  
  
  • A feature of major version M > N can depend on feature version N if it doesn't include breaking changes (In the current state of affairs, I expect that there would never be breaking changes)
  
  

I expect that feature assemblies will never be referenced by concrete runtime, library or application assemblies.

For the rest of this proposal, I will address the proposed feature (sorry 😑) as feature:
A feature is either provided or depended upon. Usage of any other term is likely a mistake on my part.

Wiring up features

Within libraries

A new TFM features is created, acting similarly to netstandard TFMs.
Library projects can target the TFM features like they would target .NET Standard, .NET FX, .NET Core or other TFMs.

When targeting the TFM features, the library project must reference all features it depends on:

• We could assume that the TFM features provide zero API by default. (But maybe it provide some kind of minimal subset ?)
• Features could be bundled in larger feature packages to avoid referencing hundreds of feature packages. (Similar to Microsoft.AspNetCore.All)
• The project can reference other libraries that are based on the features TFM, and will transitively inherit feature dependencies.

At build time, the toochain will load all feature assemblies and construct a model of the global feature set required by the library by merging all type definitions in the feature assemblies.
This can be done in the compiler or before calling the compiler, via an external tool. (I feel it might be easier to add this in the MSBuild build process than in Roslyn.)

All metadata references to features would be mapped to a non-existant and well-known features assembly (possibly signed, and in that case, the signing key would have to be public).

⚠️ Important: The feature dependencies still need to be stored somewhere inside the resulting assembly… But I don't know how they should be stored. (Custom attributes, or regular .NET assembly references ?)

Within applications

What is true today will still hold true with features: Applications are expected to target a concrete framework.
However, application projects would gain the ability to reference features-based projects and libraries.

When an application references a features-based project, the toolchain (NuGet & MSBuild) must always validate that the current TFMs for the project support all of the required features, directly, or via compatibility NuGet packages.

At build time, the toochain will load all feature assemblies indirectly referenced by the project and construct a shim assembly covering all of the required features:

• This assembly will have the well-known name features (identical to what is referenced by feature-based library projects)
• This assembly will contain TypeForwardedToAttribute attributes for each and every type referenced by the features. (That's assuming that feature support has already verified for the project, so that all the members of forwarded types are guaranteed to exist)
• The assembly is (likely) signed with the well-known signing key.

The generated features shim assembly will be bundled with the application, and act as the glue between features-based libraries and the underlying framework. (This should be very similar to how .NET Standard works, if I'm not mistaken)

ℹ️ NB: In the end, what I propose is that, rather than the runtime(s), the application is responsible for providing (wire-up) the features that the libraries depend upon. (But that this is handled by the toolchain)

Providing features (i.e. Concrete implementations)

Splitting type definitions across multiple reference-like assemblies should not break typical .NET expectations:

• A concrete type implementation can only be provided by one assembly
• Where the concrete type implementation ends up being (i.e. in which assembly) is dependent on the runtime, and should not matter to the library developer
• The runtime is never aware of the partial type definitions that were used to build libraries

There must exist a mapping between runtimes/frameworks and features, and features could likely be always provided via NuGet packages.
This would provide a single source of metadata for the toolchain to generate its shim assemblies.

This is basically what we could expect:

• A NuGet package can declare a list of implemented features.
• If a NuGet package claims to provide a feature, it must provide this feature entirely, even if that means that some method would throw NotImplementedException
  
  
  
  • Implementation of the feature could, however, rely on other dependencies (via TypeForwardedToAttribute)
  
  
  • Implementation of a feature is always relative to a valid TFM that is not features (but it could be netstandard2.0)
  
  
• For each framework, a NuGet package with the correct Major.Minor version would exist. (e.g. Framework.NETCoreApp, Version=2.1.0)
  
  
  
  • Framework NuGet packages would not contain the framework itself
  
  
  • Framework NuGet packages would define which features the framework supports, and how they are mapped
  
  
  • Framework NuGet packages could be updated incrementally to add features as they are standardized
  
  
  • Framework NuGet packages would be flagged specifically
  
  

Features on the NuGet side

All of the above requirements would likely require adaptations on the NuGet side:

• As said earlier, Microsoft or .NET Foundation should reserve a NuGet prefix for features (e.g. Feature.) to ease discovery of features and avoid pollution
• Microsoft or .NET Foundation should reserve a NuGet prefix for frameworks (e.g. Framework.) to ease discovery of framework-feature mappings
• A NuGet package must be identifiable as a feature
  
  
  
  • features are to be always considered framework agnostic
  
  
  • feature packages should only contain one reference assembly (Named the same as the feature ?)
  
  
  • feature packages should be ignored by older NuGet implementations
  
  
• A NuGet package must be able to depend on a feature
  
  
  
  • That might be as simple as declaring a regular package dependency
  
  
• A NuGet package must be able to provide a feature
  
  
  
  • The features which are provided must be listed in the package… (How ?)
  
  
  • A NuGet package implementing one ore more feature must contain, for each TFM where it applies, exactly one well-known assembly named features, signed with a well-known key, and containing/forwarding the implementation of all the features.
  
  
• A NuGet package must be identifiable as a framework
  
  
  
  • Framework NuGet packages are not considered framework agnostic (they… are… applying to themselves, somehow)
  
  
  • Framework NuGet packages follow the same rules as NuGet packages that provide features
  
  

Remarks

How should features be sliced from .NET Standard ?

The goal of this proposal is not (yet) to propose how features should be sliced, but to propose how to allow them to be sliced.
However, it must be noted, that there are many ways to slice an API with the feature proposed here.
While most types will not require any special treatment, some types may be better sliced into multiple separate features, rather than exported as a single feature.

We could consider, for example, that the Stream class could be exported in a Feature.IO.Stream.Sync and Feature.IO.Stream.Async, with possibly even a common feature Feature.IO.Stream.Core.

I'm sure that would generate a lot of discussions anyway… 🙂

Possible uses of features

Apart from the obvious modularization of the standard, features could also be used in the following scenarios:

• For framework dependencies of a .NET language such as C#: The feature for the language would be versioned in parallel with the language version, and, for each version, declare every type and member needed to fully support that version of the language.
• For defining an optional standardized .NET UI API. (Whose implementation might be tied to the runtime)

• Defining a new feature over a third party API, for allowing developers to more easily swap the implementation (because features apply to any .NET type, and not just only interfaces)

  Finding how features are provided

  While features provided by the runtime can be easy to discover (the user only would only need to import the corresponding package, and the tools might even do it automatically), it might be much harder to find that a package for supporting a feature exists on a given runtime.

  Given that the responsibility for ensuring that à feature is provided is delegated to the final user (user of the library(ies) depending on features), there might be a need for some sort of mechanism helping to discover feature implementations.

  It would be great if NuGet coult auto import default feature-providing packages, based on some kind of repository… So that everything goes smoothly on the end-user side.

(Supposed) pros and cons of this proposal

Pros

• More fine grained control over dependencies (looser coupling, maybe recovering some of the possibilities that were lost by removing Assembly Neutral Interfaces)
• Ability to standardize small parts of the API surface rather than a whole framework (e.g. new Span APIs could be standardized faster on their own)
• Ability to make some newer APIs compatible with old .NET versions again (That is an important scenario in some migration paths)
• Ability to standardize things such as UI APIs (e.g. Windows Forms) without redefining ".NET Standard"
• This proposal does not require modifying existing runtimes

Cons

• This proposal adds yet another layer of complexity to the already complicated world of framework compatibility
• Consuming libraries implemented over features requires upgrading the toolchain (that is not a problem for everyone, but it can be for some)
• This proposal may requires non-trivial additions to NuGet (let's hope not)
• Slicing the .NET Standard API in a potentially large number of fine-grained features must be done very carefully, and may take quite a bit of time

Conclusion

Thank you very much for reading this to the end !

I'm waiting for your feedback. 😉

[davidfowl]

Having lived through DNX, PCLs and .NET Core 1.1 until now. This is extremely complicated and I don't think you can get it right. I've never seen any other platform attempt to solve this. Everyone picks some stuff to go in their "std lib" and it's mostly fine. You deal with platform warts over time and move on. I think that pattern repeats itself not because it's perfect but because it's easy to reason about.

I don't have any concrete feedback for this proposal as yet but my knee jerk reaction having lived through something that similarly tried to solve all of the problems didn't work so well. Maybe I'm missing the "ah hah" reason why this is better.

[ahives]

Maybe this is a simplistic view on my behalf but, can't you simply target multiple frameworks? I have done this in my OSS projects and it works fine. This proposed solution if implemented would make the framework much more complicated. Think about what this would mean to all the consumers of this, that is, upgrading CI/CD to deal with versioning, forcing all future applications to programmatically deal with the new versioning scheme in code, etc.

To me, the key is to not get stuck on a particular framework version. Case in point, we developed an internal library 5 years ago and decided to run the .NET Core migration tool against it a few months back and it had a 99.x% compatibility years before .NET Core was released. Of the 100k lines of code there was 1 line that was not compatible.

I feel this proposal doesn't solve the problem of framework lock in and attempts to bend the standard for backward compatibility with a version of the framework that is out of support.

I don't understand your comment when you say, "But I do also perfectly understand the concerns about .NET Core 2.1 being the only implementation already ahead in the standard. And because of that I do fear that .NET Standard 2.0 will be the last .NET Standard version ever."

Why in your view would .NET Standard 2.0 be the last .NET Standard version ever? The framework is being updated in parallel of the standard so I don't understand where this fear is coming from. Also, based on the feedback from customers, I am pretty sure Microsoft will be continuing .NET Standard releases well into the future. There are a lot of us .NET developers that do not want to target Windows so...

Also, you said, "...the fact that .NET Standard 2.0 is not compatible with .NET 4.5 can sometimes be an obstacle in the migration process to .NET Core. (That's something we are faced with everyday in my current job)"

I think a lot of people face the same "problem" as do I, but one thing to remember is that .NET 4.5 is no longer a supported version of the framework (its been 6 years). I for one do not want .NET Core to be bound by 4.5. I have the same issue with a good amount of code at my job and the solution is to iterate toward .NET Core/Standard and not try to bend it to our will. Instead of jumping straight to .NET Core/Standard I suggest perhaps moving to Framework 4.7.2 and then .NET Core/Standard.

Hope I wasn't being a jerk but I think you are facing a legitimate issue but I just strongly disagree with approaching it from radically changing the standard. I think this problem should be solved by applications using .NET and not the framework. That said, I am open for convincing. :)

[Petermarcu]

PCL v2 was basically built on top of the idea of many different smaller "features" that could be composed. Any platform would say which features they had and the tooling would limit you to used the API's in the features that were supported by the intersection of the platforms you wanted to run on. Those "features" were called contracts. This sounds a lot like what PCL v2 was.

The problem was that it encouraged and enabled .NET implementations to fragment the ecosystem and created a lot of complexity. The .NET Standard approach is aimed to simplify and create a solid foundation for the ecosystem.

[stakx]

Great proposal, thank you for writing it up!

A system based on distinct feature sets (which may overlap) would be a great improvement over .NET Standard's purely additive "catch up or die" onion layer model.

A few random points that I thought about while reading it:

This assembly will contain TypeForwardedToAttribute attributes for each and every type referenced by the features.

What about [TypeForwardedFrom]? The implementing type would ideally be marked with such an attribute such that it's possible to find out (via reflection) the type's "logical" (feature-based) name. This could be important e.g. for AssemblyBuilder.Save: Dynamically generated assemblies should reference the original, "logical" type (such as [mscorlib]System.Span`1) and not the implementing type ([System.Private.CoreLib]System.Span`1). AFAIK .NET Core currently places fixed [TypeForwardedFrom] on types in the runtime pointing to shim DLLs. That would no longer work here because the shim DLLs are generated on the fly and not known beforehand.

A NuGet package can declare a list of implemented features.

The features which are provided must be listed in the package… (How ?)

This declaration should happen at the assembly level and not at the NuGet level IMO, otherwise you end up with a situation where you cannot compile unless the build tools have NuGet integration. That feels somewhat unwise to me. It should be possible to compile with nothing other than source code and (reference) assemblies to keep the whole ecosystem more "grounded".

This proposal adds yet another layer of complexity to the already complicated world of framework compatibility

... and this may well be the main con. I agree with @davidfowl that it might not play out well to try once more solving this problem that might be very, very hard to get right for everyone. Are we ready for another year-long treat of assembly binding redirect problems showing up all over the place?

That being said, thanks again for coming up with the proposal. The fundamental idea behind it seems like a very good one to me.

[DamienDennehy]

I totally get where you're coming from but I feel that it puts us back 5 years (or more) to the PCL days.

Span Standard 1.0 is giving me Profile259 flashbacks.

[dcuccia]

Carefully reasoned proposal, but I agree that this would go backwards and diminish the stability of the ecosystem. Less convinced that we shouldn't have a leading standard and light some fire under encourage the .NET vNext team to implement Span.

[terrajobst]

@GoldenCrystal

First of all thanks a ton for the detailed write-up! I find myself in the same as @davidfowl that my knee-jerk reaction is: thanks, but no thanks. Been there, done that, and no desire to go back.

Let me summarize what @davidfowl and @Petermarcu said and add to it:

1. Splitting the standard into smaller feature sets is only meaningful if implementations can pick and choose which features to provide.
2. This causes fragmentation and makes it harder for customers to reason about which features they can depend on in order to run on future implementations.
3. We've already tried this with PCLs (profiles) and .NET Core v1 (features called contracts). It caused a lot of challenges in tooling and modelling.
4. In principle this can be modeled, but it's very hard to do as a continuation of the existing .NET platform as providing certain compatibility guarantees is hard.

I do fear that .NET Standard 2.0 will be the last .NET Standard version ever

Given the 30+ PRs and planning documents for vNext I hope we've eliminated that concern by now :-)

Aside of that, the fact that .NET Standard 2.0 is not compatible with .NET 4.5 can sometimes be an obstacle in the migration process to .NET Core.

That is true but I don't think your proposal addresses that. I believe multi-targeting (i.e. having a single project build for both netstandardxx and net45) will solve that today.

So I wondered, what if, instead of having one unique standard, there were many smaller standards that could be implemented à-la-carte by runtimes/frameworks/NuGet?

That sounds appealing in principle. I've personally worked on & designed features around this premise for 5+ years. Spoiler alert: it didn't work well :-)

But, I assure you, I am not proposing to reiterate the exact same mistakes as in .NET Standard 1 era. (Which is not to say that I'm not taking strong inspiration on what has/had already been done. 😉)

I don't see how your proposal would address the conceptual challenges, i.e. I don't believe the problems were the implementation but the inherently tied to the fact that fragmentation occurs. Your notion of features is virtually identical to our contracts. Those were just regular assemblies, but only contained API surface and the implementation could have any factoring it wanted. Feature dependencies are just assembly references, which, you guessed, just fall out.

But it doesn't solve the problems around fragmentation, modeling the set, and making referencing the platform not excessively complicated for developers.

If you primary concern is that you can't target existing platforms with the standard, I believe we've a much better answer today in tooling via NuGet packages and multi-targeting. And future platforms simply support .NET Standard directly.

So while I appreciate your ideas and enthusiasm, I believe we (the .NET team) have explored this route extensively over a long period of time. Thus, I consider this idea as researched and deemed to be unworkable.