Machinelearning: Separate Nuget versioning for "final" and "experimental" nugets

[TomFinley]

Right now all our nugets have one global version. Once we ship V1, however, there will be some assemblies and nugets that are "final," in that their API surface is locked and going forward we do not make breaking changes, and other assemblies doing more experimental work where we're not quite so sure what the final surface will look like.

An example of this is Microsoft.ML.StaticPipe and friends. The statically typed pipelines work is interesting and we want to continue working on it post v1. Similarly, Microsoft.ML.Ensemble seems like something we don't have time to get the API "right" for yet. (Though we might solve that specific one in a different way by just hiding it altogether for now.)

A different example of this would be things like Microsoft.ML.EntryPoints, which we have to make public for "reasons," but cannot reasonably be said to have a public API.

Another example would be some sort of "smuggler's cove" assembly/nuget (which does not yet exist) that exposes some infrastructure that some important scenario or other needs access to, but we are not comfortable making as part of our public API. (E.g., some conduit to "get at" the internal methods of the ComponentCatalog.)

@eerhardt opined that if we do that, then of course there should be a separate versioning scheme for those. This would include changes to the package build infrastructure to accomodate this "two tier" structure we do not have yet.

[shauheen]

smuggler's cove :)

[eerhardt]

@maryamariyan will add the build infrastructure in order to have 2 groups of NuGet packages: "stable" and "experimental".

We still need to specify which packages will be "stable", but we can mark those packages after the build infrastructure is there.

[Ivanidzo4ka]

So here is list of Stable and Unstable:

Stable:
Microsoft.ML
Microsoft.ML.CpuMath
Microsoft.ML.HalLearners
Microsoft.ML.ImageAnalytcs
Microsoft.ML.LightGBM
Microsoft.ML.Mkl.Redist

Unstable:

Microsoft.ML.DnnImageFeaturizer.AlexNet
Microsoft.ML.DnnImageFeaturizer.ResNet101
Microsoft.ML.DnnImageFeaturizer.ResNet18
Microsoft.ML.DnnImageFeaturizer.ResNet50
Microsoft.ML.EntryPoints
Microsoft.ML.OnnxConverter
Microsoft.ML.OnnxTransformer
Microsoft.ML.Parquet
Microsoft.ML.Recommender
Microsoft.ML.SampleUtils
Microsoft.ML.StaticPipe
Microsoft.ML.TensorFlow
Microsoft.ML.TensorFlow.Redist
Microsoft.ML.TimeSeries
Microsoft.ML.Ensemble

[maryamariyan]

@Ivanidzo4ka from the list above (https://github.com/dotnet/machinelearning/issues/2279#issuecomment-467614084) I couldn't find csproj for the following in the repo:

• Microsoft.ML.Mkl.Redist
• Microsoft.ML.TensorFlow.Redist

Are they under a different name?

[Ivanidzo4ka]

I believe @sfilipi and @zeahmed can provide you context.

[sfilipi]

@maryamariyan look under the src/Redist. They are .proj files

[maryamariyan]

Thanks @sfilipi
I found Microsoft.ML.TensorFlow.Redist under src/Redist. But still not found Microsoft.ML.Mkl.Redist

[sfilipi]

@maryamariyan sorry, Mkl is with the other ones: https://github.com/dotnet/machinelearning/blob/master/pkg/Microsoft.ML.Mkl.Redist/Microsoft.ML.Mkl.Redist.nupkgproj

[eerhardt]

@maryamariyan - a thing that may help here:

Our .csproj files in this project produce the managed .dll assemblies. But they don't create NuGet packages.
Instead, all the NuGet packages are created by projects that end with .nupkgproj. We call dotnet pack on these .nupkgproj files, which creates the .nupkgs.

The reason for this is that we don't have a 1:1 mapping between managed .dll projects and .nupkgs. Also, we have native/C++ assemblies that need to be packaged into the .nupkg.

So all Nuget packages are created from projects under:

https://github.com/dotnet/machinelearning/tree/master/pkg.

So for both our managed assemblies and our NuGets, we will have to version the "experimental" packages differently than the "stable" packages. Our experimental managed assemblies shouldn't have "1.0" AssemblyVersions, which is controlled by the Version properties. Here is a decent explanation about Version properties. So both the .nupkgproj and the .csproj will need to know if it should be considered "stable" or "experimental".