Runtime: COM Interop not working on Linux builds

[danwalmsley]

I noticed this.. and got the exception...

https://github.com/dotnet/coreclr/blob/master/src/mscorlib/src/System/RtType.cs#L4885

Will this be implemented in netcore 2.0?

current windows builds this works fine.

[yizhang82]

@danwalmsley Sorry. COM interop is not supported on linux (hopefully shouldn't be surprising). Typically people who asks for these are porting their existing COM code to linux or working with CoreCLR COM-based API (such as ICorDebug). Is that what you are doing?

[danwalmsley]

Im trying to integrate the dotnet core debugger, ICorDebug
Surely it must be possible to use this api on Linux?

[yizhang82]

@danwalmsley You can use C/C++ to do v-table calls on ICorDebug. If you have more questions, @noahfalk can point you to the right direction.

[danwalmsley]

@yizhang82 thanks for the pointer 😀
Is the a plan to bring com Interop even in a limited form?

Mono already supports this I believe.

[danwalmsley]

@kekekeks

[yizhang82]

@danwalmsley We don't have concrete plans yet. If we see strong feedback from community there is a good chance we might bring it back in CoreCLR, or introduce code generator that can spits out wrapper code. Supporting it in CoreCLR is not trivial as we need to properly #ifdef out the Windows-specific part (such as activation, apartments, etc) and only support the "core" COM part (ref counting, QI, vtable dispatch), but is certainly doable. I haven't looked at Mono's support but I can imagine it's probably very similar. One interesting possible use case that I can think of is to use "core" COM as object-oriented component ABI (kinda like what ICorDebug already does).

[danwalmsley]

@yizhang82 the code I am using to interface with ICorDebug compiles and runs on mono successfully (I think), do you think I could somehow compile that assembly on mono and include somehow monos com support, then call the mono assembly from netcore2.0?

[kekekeks]

I haven't looked at Mono's support but I can imagine it's probably very similar

If I remember correctly, Mono has support for COM, but not ActiveX or OLE, so one can consume anything that's based on IUnknown/IDispatch mostly without issues. It does, however, have issues with implementing COM interfaces in .NET objects, ref-counting is somewhat broken and CCWs passed to native code get garbage collected while still having references. Performance is also not that great, unfortunately.

The feature is very nice to have, since it's quite easy to implement IUnknown in portable way (it will use cdecl instead of stdcall, but that shouldn't be an issue).

[yizhang82]

@danwalmsley You can't run the mono assembly using CoreCLR because that means you are running on CoreCLR and you still don't have COM interop feature. You need to run CoreCLR and mono in the same process, and they have to communicate with well-defined ABI, such as C. It's doable, but you need to write your own glue code, and you can't pass objects around (since a Mono object is not a CoreCLR object) and you need to wrap them as opaque handles, which is a lot of hassle.
@kekekeks I agree it's good to have, but I don't know if this would be interesting to the broad .NET Core community. If there are a lot of people want to port their existing COM code to non-Windows and consume in .NET Core, that's certainly going to be very interesting. Right now I don't see that happening yet.

[danwalmsley]

@yizhang82 we never heard anything from @noahfalk, I would be interested to see more information on vtable calling.

[noahfalk]

Apologies I missed the callout earlier. I think the request for direction was more around how can you interop with an API built on vtables vs. anything specific about the semantics of using ICorDebug? @yizhang82 is the expert when it comes to interop techniques, but the two options that I know of to use native vtables from C# would be:

1) Create a flat C wrapper, then use C# to p/invoke to the C functions.
2) Use Marshal.GetDelegateForFunctionPointer() to create delegates for each function pointer in the vtable. Marshal.ReadIntPtr() can read the pointers from native memory. For a given pointer ICorDebugFoo* p in native code you could transform it into managed with something like:

     IntPtr p = GetTheICorDebugObjectPointer();
     IntPtr vtablePtr = Marshal.ReadIntPtr(p);
     Func<IntPtr,int> addRefFunction = Marshal.GetDelegateForFunctionPointer<Func<IntPtr,int>>
     (Marshal.ReadIntPtr(vtablePtr + 0));
     // marshal other vtable slots as needed...

     addRefFunction(p); // calling p.AddRef();

There might be some mistakes in there as I did it on the fly, but hopefully it conveys the right idea. Its likely you'd want to cache the delegates for different vtables you encounter and share them for objects of similar types, so the complexity can grow a bit. Effectively you are building an alternate C# implementation of the portion of COM-interop that deals with vtables and reference counting.

[Const-me]

@yizhang82

If there are a lot of people want to port their existing COM code to non-Windows and consume in .NET Core

The feature is not just about porting existing code. IUnknown-based interface is a straightforward ABI that allows to expose any OO API implemented in native code.

Take a look at Microsoft’s Direct3D or MediaFoundation. They don’t use COM but do expose IUnknown-derived interfaces. The pattern is very useful for general purpose stuff. Instead, for .NET on Linux, I currently have to manually wrap my objects into exported C functions like class_method(class *pThis).

[yizhang82]

@Const-me I agree with you that COM could be a very viable ABI for wrapping objects. If you only keep the very basics - just IUnknown, and leave out other stuff (such as activation, apartments, security, etc), it's actually a very reasonable option. As a matter of fact, when people port their existing COM code to other platforms, they do exactly that. Unfortunately such practice is not very common in the open source community (the class_method(class_handle this) way is the standard pattern), so we went a different route - using AOT / MCG to support such scenarios. On retrospective, it's hard to say whether we made the right call at that time.

Disclaimer: I don't work in .NET team anymore so this is just my personal opinion.

[tannergooding]

The COM interop, ignoring all of the other cruft (threading, security, proxies, etc), is really just a nice way to marshal the VTBL of a native class.

That is, when you declare an interface:
```C#
interface IUnknown
{
int QueryInterface(in Guid riid, out IUnknown ppvObject);
int AddRef();
int Release();
}

This is really just shorthand for:

```C#
struct IUnknown
{
    public IUnknownVtbl* lpVtbl;

    public delegate int QueryInterface(IUnknown* pThis, in Guid riid, out IUnknown ppvObject);
    public delegate int AddRef(IUnknown* pThis);
    public delegate int Release(IUnknown* pThis);
}

struct IUnknownVtbl
{
    public IntPtr QueryInterface;
    public IntPtr AddRef;
    public IntPtr Release;
}

It also means that you can directly invoke the method table entries, rather than needing to manually call Marshal.GetDelegateForFunctionPointer on the vtbl entries.

It also means, since it is a COM object, it is wrapped in a COM callable wrapper and the runtime will track the lifetime of the object and call Release for you when all references go away.

[tannergooding]

I opened an issue a while back (https://github.com/dotnet/coreclr/issues/6531) suggesting that CoreCLR provide a new interop type (Native) which would ignore all of the COM cruft and just marshal the VTBL.

For example, it could be used for a hypothetical interface ICalculator (which doesn't use COM)
```C#
interface ICalculator
{
float Add(float left, float right);
float Sub(float left, float right);
float Mul(float left, float right);
float Div(float left, float right);
}

Which saves you from manually doing:
```C#
struct ICalculator
{
    public ICalculatorVtbl* lpVtbl;

    public delegate float Add(IUnknown* pThis, float left, float right);
    public delegate float Sub(IUnknown* pThis, float left, float right);
    public delegate float Mul(IUnknown* pThis, float left, float right);
    public delegate float Div(IUnknown* pThis, float left, float right);
}

struct ICalculatorVtbl
{
    public IntPtr Add;
    public IntPtr Sub;
    public IntPtr Mul;
    public IntPtr Div;
}

It could also be used for a COM compatible interface, provided the user manually managed the lifetime of the object.

[AndreyAkinshin]

We really need the COM Interop support in Rider. It's a show-stopper which prevents migration from Mono to .NET Core.
@karelz, what the current plans for this feature?

[ro-jo]

I just wrote my first code for .Net Core 2.0 and found this thread after disappointment with the purported cross platform capability.

I made a .Net Core DLL with one native entry point to return an IUnknown interface, exporting a simple and extensible api as an interface for C++ apps to use new code.

It doesn't work. The .Net Core runtime fails on Linux, and works on Windows: When I try to MarshalAs IUnknown, .Net Core throws an exception on Linux, but works fine on Windows.

I had a look through the .Net Core 2.0 docs, e.g.
https://docs.microsoft.com/en-gb/dotnet/api/system.runtime.interopservices.unmanagedtype?view=netcore-2.0
It talks a lot about COM types without mentioning it's not cross platform.

Even without a full COM implementation on Linux, basic support for marshaling interfaces would provide developers with a decent way to create their own interop modules.

[kekekeks]

@ro-jo The tooling from SharpDX works fine with COM interop on Linux/OSX. AvalonStudio is currently using it for libdbgshim.so.

It won't be as seamless as it is with the built-in COM support, but it works and it's faster than built-in one.

[ro-jo]

@kekekeks - thanks for suggestion, SharpDX looks interesting, but it's a bit like linking to MFC just to use the String class. I'm interested in built-in cross-platform marshalling support. My C++ code is not COM, it doesn't use any COM infrastructure. In Linux I just mocked up IUnknown. IUnknown is a good way for .Net Core to support integration with C++ and other native languages without the need for ugly code on both sides, a heavy infrastructure, or unwieldy 3rd party dependencies. I don't want to reference any COM module in .Net Core, it isn't platform independent, I don't expect that at all - but I see nothing wrong in writing such code for my own applications on any platform, that can talk to .Net Core in a reasonable manner. That way, anyone that does want to add COM support can create a private interop, and people like me can integrate new .Net Core pieces into old applications without drudgery. As it stands, I'm faced with creating a plethora of extern "C" type entry points on both sides with loads of calls to create delegates, complicated PInvoke attributes, and so on. No problem, you can't have everything.

Wouldn't it be nice if this worked cross platform? To do this, no support for COM required from operating system:

In C#:

using System;
using System.Runtime.InteropServices;

namespace HelloWorld.Interop
{
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown), 
     Guid("1157BDDC-887C-46BC-BDA0-427B05BE1A0B")]
    public interface IHelloWorld
    {
        void PrintString();
        int PrintString2([MarshalAs(UnmanagedType.LPWStr)] string msg);
    }
}

In C++:

// No COM headers required, assume I've typedef'd IID, HRESULT , ULONG, STDMETHODCALLTYPE, etc.
// Need care with string ownership etc. there will be no CoTaskMemFree.
// If C++ creates an instance, it manages lifetime, C# uses IUnknown.
// If C# creates instance, C# manages lifetime, C++ uses IUnknown.
struct UUIDS
{
    static constexpr const IID IID_IUnknown = { 0L, 0, 0, 0xC0,0,0,0,0,0,0,0x46 };
    static constexpr const IID IID_IHelloWorld = { 0x1157BDDCL, 0x887C, 0x46BC, 0xBD, 0xA0, 0x42, 0x7B, 0x05, 0xBE, 0x1A, 0x0B };
};

struct IUnknown
{
    virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, void **ppObject) = 0;
    virtual ULONG STDMETHODCALLTYPE AddRef( void) = 0;
    virtual ULONG STDMETHODCALLTYPE Release( void) = 0;
};

struct IHelloWorld : public IUnknown
{
    virtual HRESULT STDMETHODCALLTYPE PrintString() = 0;
    virtual HRESULT STDMETHODCALLTYPE PrintString2(const char16_t *ptr, int32_t *pResult) = 0;
};

@tannergooding I looked at your issue dotnet/runtime#6413, I totally agree with the idea, and it was closed because not just any interface can interoperate with .Net - I also totally agree with that. But if .Net Core doesn't support COM, maybe a new interop type of 'CorCOM' would fare better.

[jkotas]

it's a bit like linking to MFC just to use the String class

Why do you think so? SharpGenTools is a tool that you run at built time. It is not a big ugly library that you link with.

I'm interested in built-in cross-platform marshalling support

There are many different ways to do interop and marshaling. There is no reasonably small one size fits all. We have COM and WinRT interop on Windows built into the runtime for historic reasons and backwards compatibility. It is intertwined with the runtime that makes it hard to maintain and evolve. Everybody program pays for it regardless of whether it is actually needed.

We do not plan to have more builtin interop like that. We believe that the right way to support interop with other language environments (C++, Java, Objective C, Python, Go, ...) is via indendent tools like SharpGenTools.

My C++ code is not COM, it doesn't use any COM infrastructure

If your code is C++, you may want to look at tools like https://github.com/mono/CppSharp to generate your interop glue and not do unnatural things to go via COM. (@jeffschwMSFT I would be useful to have an example on how to use CppSharp with .NET Core.)

[Const-me]

@jkotas

https://github.com/mono/CppSharp to generate your interop glue and not do unnatural things to go via COM

That thing looks very fragile. It exposes not just API, but a huge part of C++ implementation as well. And unlike C, C++ has no standardized ABI.

vector, map, set support is experimental. The rest of the containers, both standard and custom, are unsupported at all. These things are essential in modern C++, e.g. that very CppSharp project uses unordered_set, unordered_map, and unique_ptr template containers.

Now compare to IUnknown-based interop. The ABI is rock solid and can be used across compilers and even across languages. Unlike C++ exceptions, HRESULT return codes are very portable. Class memory layout and memory management remain internal C++ implementation details, I can change these as I see fit and I don’t even need to re-compile C# code afterwards as they don’t affect the ABI.

[ro-jo]

@jkotas I see you and @kekekeks are a team. In the order that you picked out quotes from my comment:

1. I didn't say 'big ugly library' did I? It's a simile implying inappropriate use of something. I aim to minimise additional dependencies and build steps. The more you add, the greater the probability something will go wrong down the line. When the gain is small, it's counterproductive.

2. I'm not asking for COM interop, I tried to make it clear. I hoped .Net Core was able to marshal IUnknown interfaces, it seems like a natural fit to me, using well known constructs in each language. But facing such strong reaction from you, I can tell you I don't care as much as you think, and am more than happy to abandon it. You mention there are the many different ways to do interop for C++, I can only think of one: CreateDelegate (C++) / PInvoke (C#). Please let me know the others...
   (I'm not interested in varieties of marshalling or other well known topics from the .Net Framework - but it might help if you published a list of what .Net Core cannot marshal. By the way, why have you added backward compatibility in Windows for IUnknown? What historic reasons make it more compelling in Windows, why can't we just use .Net Framework instead? How does it fit the cross-platform strategy?)

3. I wasn't asking for help or recommendations, but I'm familiar. "This tool allows you to generate .NET bindings that wrap C/C++ code allowing interoperability with managed languages". Does it also write C++ bindings that wrap .Net Core code? For example, if I have a native codebase of 1,000,000+ lines, and want to add a new .Net Core component, I'd prefer to write my own code to interface with it. As it stands, it's a lot more complicated with .Net Core than using C++/cli.

[tannergooding]

Now compare to IUnknown-based interop. The ABI is rock solid and can be used across compilers and even across languages.

I agree.

However, the current COM interop on Windows comes with a performance penalty (which can be non-trivial), as will any other marshalling that converts a managed construct (such as a class, interface, or delegate) into some native wrapper.

I think the primary issue with the alternative (to adding cross-platform COM marshalling support to the runtime) is that it is rather cumbersome and still requires some amount of marshalling overhead (the alternative is manually creating a VTBL struct, conforming to the COM ABI).

However, C# is adding a few features that would make this significantly easier:

• Support for an 'unmanaged' constraint
  
  
  
  • This is currently milestoned for the next release (currently that is 7.3)
  
  
  • This allows using pointers with generics (T*, where T : unmanaged)
  
  
• Support for 'static delegates'
  
  
  
  • This is currently milestoned for the next major version (currently that is 8.0), but is dependent on some design decisions on how they work with AppDomains
  
  
  • This allows a blittable delegate type that is basically equatable to an unmanaged function pointer
  
  

Both of these would significantly improve the experience and allow it to remain performant/blittable (no marshalling overhead, as the raw bits are directly copied).

As a user of these features you could just do something like:
```C#
// Reusable struct to wrap the vtbl
struct NativeComObject where T : unmanaged
{
public T* lpVtbl;
}

struct ICalculatorVtbl
{
public static delegate float _Add(ICalculatorVtbl* pThis, float left, float right);
public static delegate float _Sub(ICalculatorVtbl* pThis, float left, float right);
public static delegate float _Mul(ICalculatorVtbl* pThis, float left, float right);
public static delegate float _Div(ICalculatorVtbl* pThis, float left, float right);

public _Add Add;
public _Sub Sub;
public _Mul Mul;
public _Div Div;

}
```

This removes the need to create a struct to hold the Vtbl, per native type.
This removes the need to call Marshal.GetDelegateForFunctionPtr (and the need to cache it)
This removes all of the marshalling overhead
Just as with regular COM, you only need to rerun your generation tool when the native contract changes (from Windows COM, this was generally running midl and tlbimp)

[ro-jo]

@tannergooding I wouldn't like to see that in my production code, and also, what about inheritance? This is C#, and we're stooping to code that looks worse than 'C', it's supposed to be the other way round!
And I don't want to have to mark assemblies as 'unsafe' either - isn't that the case with C# pointers?

[Const-me]

@tannergooding Before today I didn’t knew about SharpGenTools. Unlike CppSharp, IMO it looks good. I’ll give it a shot (BTW doing some embedded work for ARM Linux that has both .NET and C++ components), maybe it’s good enough already.

On Windows, I don’t use generation tools nor type libraries, these are IMO way too complex and only required for “true” COM.

I manually write interfaces in both languages, using __interface __declspec( uuid( "…" ) ) i: public IUnknown in C++, and [Guid( "…" ), InterfaceType( ComInterfaceType.InterfaceIsIUnknown )] interface i in C#. Your proposed approach makes it harder, i.e. I when I change an interface method, I need to update 2 places of C# instead of 1. Also, unlike C# interface, that vtbl structure doesn’t look idiomatic C#.

I don’t need marshalling either. To avoid marshalling, I have a C++ base class for my objects that inherits from CComObjectRootEx and implements IMarshal interface using CoCreateFreeThreadedMarshaler API, this instructs .NET to call my objects directly without any marshalling.

But what I do need is IUnknown support in the runtime or libraries, namely reference counting, and in rare cases casts to another interface (that calls QueryInterface under the hood). As far as I understood from your example code, your ICalculator interface doesn’t inherit from IUnknown, which means the lifetime management, nor casts, will be available.