Cntk: Guidance on extending managed wrappers e.g. for C#

[nietras]

I would like to know if there is any guidance on if one wants to expand extend the SWIG generated code either by changing SWIG interface files but also if adding supplemental code e.g. if I wanted to add a way to create Value from unmanaged memory, without Value owning the memory, what is the procedure?

I could add a Value.Manual.cs file with code extending the Value.cs SWIG generated file since Value is a public partial class, which is good.

There are a lot of opportunities to improve the managed interface, due to the less than ideal code generated by SWIG. Both from a user and performance perspective.

[davidbrownellWork]

I agree that the code SWIG generates isn't as CSharp-y as we would like.

We are planning to take a bigger swing at the C# API in the near future to make it something closer to what one would expect a C# API to look and feel like. With that in mind, I'd recommend submitting pull requests for changes that would make your immediate experience better; however, for the sake of your time, I'd recommend not making a large investment in platform-related changes.

[nietras]

We are planning to take a bigger swing at the C# API in the near future

@davidbrownellMS sounds great. Actually we have been working on what we call CntkExtensions for this exact reason and we would be happy to share this with you (all the code) and give our thoughts on this. Perhaps, this would be better continued by email?

Our API is a work in progress, our focus are CNNs and with it you can write:

// Preamble (this ensure reproducibility and easy changing of defaults)
var d = DeviceDescriptor.CPUDevice;
var f32 = DataType.Float;
const int initialRandomSeed = 879712982;
var r = new Random(initialRandomSeed);
Func<CNTKDictionary> convWeightInit = () => WeightInit.GlorotNormal(r.Next());
Func<CNTKDictionary> denseWeightInit = () => WeightInit.GlorotNormal(r.Next());
var biasInit = BiasInit.Zero();
Shape3D inputShape = (28, 28, 1);
const int outputShape = 10;

// Network definition (just an example for now)
NDShape inputNDShape = NDShape.CreateNDShape(inputShape);
Function input = Variable.InputVariable(inputNDShape, f32);
NDShape outputNDShape = NDShape.CreateNDShape(new int[] { outputShape });
Variable output = Variable.InputVariable(outputNDShape, f32);

var batchNormTimeConstant = 5000;

var network = input
    // Block 1
    .Conv2D((3, 3), 32, (1, 1), Padding.Zeros, convWeightInit(), biasInit, d, f32)
    .BatchNorm(BatchNorm.Spatial, denseWeightInit(), batchNormTimeConstant, d, f32)
    .ReLU()
    .MaxPool2D((2, 2), (2, 2), Padding.None)
    // Block 2
    .Conv2D((3, 3), 64, (1, 1), Padding.Zeros, convWeightInit(), biasInit, d, f32)
    .BatchNorm(BatchNorm.Spatial, denseWeightInit(), batchNormTimeConstant, d, f32)
    .ReLU()
    .MaxPool2D((2, 2), (2, 2), Padding.None)
    // Block 3
    .Conv2D((3, 3), 64, (1, 1), Padding.Zeros, convWeightInit(), biasInit, d, f32)
    .BatchNorm(BatchNorm.Spatial, denseWeightInit(), batchNormTimeConstant, d, f32)
    .ReLU()
    .MaxPool2D((2, 2), (2, 2), Padding.None)
    // Block 4
    .Dense(64, denseWeightInit(), biasInit, d, f32)
    .BatchNorm(BatchNorm.Regular, denseWeightInit(), batchNormTimeConstant, d, f32)
    .ReLU()
    // Block 5
    .Dropout(0.5, seed: r.Next())
    .Dense(10, denseWeightInit(), biasInit, d, f32);

var loss = CrossEntropyWithSoftmax(network.Output, output);
var accuracy = ClassificationError(network.Output, output);

I have plenty of issues with existing Python APIs and we believe a much better API can be created in C# using modern language features and avoiding the "one line does everything" APIs that Python suffers from e.g. Convolution will have something like activation='ReLU', which is verbose, uses magic strings, and is not flexible with regards to where the activation is applied e.g. after batch normalization or even max pooling. The list goes, but I don't want this to turn into a Python rant ;)

[nietras]

@davidbrownellMS we of course also have a list of wishes with regards to the low level APIs exposed directly from CNTK, e.g.:

• Methods to create Value from native memory, as a view, without owning the memory. To avoid memory copies and unnecessary complexity. Same for creating using refs.
• Support for using new types such as Span<T> and ValueTuple<..>.

And so forth :)

[davidbrownellWork]

Excellent! I'd love to take a look at what you have put together and learn more about what you'd like to see in a C# API. Per your suggestion, let's continue the conversation offline.

Can you send me an email at david dot brownell at microsoft dot com.

[haixpham]

@nietras you can in fact do all that in Python. For example: cntk.Layers.Conv2D -> cntk.BatchNomarlization -> cntk.ReLU.

I have created a bunch of Python helper functions encapsulating those basic funtions for different usages.

The main drawback of Python API is that it is too high-level. Have to go through lots of hoops to extract internal states and weights, and to copy them to other models. Wish the CNTK team will address that problem in the future. Anyway the current support for C++, C# and UWP sets CNTK apart from other toolkits. Training code in Python and integrated inference with C++ or C#, that's the way to go.