Runtime: When calling into COM API, behavior is different when a COM exception is thrown between desktop and core

[ameltzerMSFT]

When calling into a COM API that errors in .NET Desktop, a COMException is thrown. In .NET Core, a TargetInvocationException is raised with an inner exception of COMException is thrown. This is a breaking change for any app that relies on COM API calls throwing a COMException for flow control.

Example repro:

$wilib = Join-Path (Get-Item .).FullName 'Interop.WindowsInstaller.dll' # Interop.WindowsInstaller.dll is a RCW for the WindowsIntaller automation library
$msi = Join-Path (Get-Item .).FullName 'AdvertMSI.msi' # This can be any MSI as long as it doesn't have any "ServiceInstall" parts

$src = @"
using System;
using System.Reflection;
using System.Runtime.InteropServices;

public class WIWrapper {
    public static object Repro(string path)
    {
        WindowsInstaller.Installer inst = (WindowsInstaller.Installer)Activator.CreateInstance(Type.GetTypeFromProgID("WindowsInstaller.Installer", true));
        WindowsInstaller.Database db = inst.OpenDatabase(path, 0);
        try 
        {
            WindowsInstaller.View v = db.OpenView("SELECT ServiceInstall FROM ``ServiceInstall``");
            return v;
        } 
        catch (COMException)
        {
            return "caught exception";
            throw;
        }
    }
}
"@


Import-Module $wilib
Add-Type -TypeDefinition $src -ReferencedAssemblies $wilib

[WIWrapper]::Repro($msi)

In .NET Desktop, COMException is caught and the PowerShell script writes "caught exception" to the logging stream.

In .NET Core (when using PowerShell 7.0 RC3), the exception bypasses the code entirely and is processed by the PowerShell runtime itself:

PS C:\Users\meg\Documents\powershell_repro> .\repro.ps1
MethodInvocationException: C:\Users\meg\Documents\powershell_repro\repro.ps1:31
Line |
  31 |  [WIWrapper2]::Repro($msi)
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "Repro" with "1" argument(s): "OpenView,Sql"

While the workaround is simple -- catch TargetInvocationException and look for an InnerException of COMException -- this breaks any shipped code that relies on the .NET Desktop behavior of being able to catch COMException for flow control.

[daxian-dbw]

Investigation from the PowerShell team

This is a behavior change in .NET Core comparing to .NET Desktop.
I updated the repro script a bit to print out the exception and all its inner exceptions, and the following is what I got from pwsh 7-rc.3 and Windows PowerShell respectively (updated repro script is pasted at the end):

## pwsh 7.0-rc.3 (.NET Core 3.1)
PS:2> .\repro.ps1
Exception FullName: System.Reflection.TargetInvocationException
Assembly Locatoin: F:\pscore70.6\System.Private.CoreLib.dll
Exception Message: Exception has been thrown by the target of an invocation.
----------  InnerException  ----------
Exception FullName: System.Runtime.InteropServices.COMException
Assembly Locatoin: F:\pscore70.6\System.Private.CoreLib.dll
Exception Message: OpenView,Sql
----------  InnerException  ----------

## Windows PowerShell 5.1
PS:1> .\repro.ps1
Exception FullName: System.Runtime.InteropServices.COMException
Assembly Locatoin: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscorlib.dll
Exception Message: OpenView,Sql
----------  InnerException  ----------

As shown above, the COMException is wrapped by ‘TargetInvocationException’ in .NET Core, and that’s why the ‘catch (COMException)’ doesn’t work as expected in .NET Core.

Updated repro script:

$wilib = Join-Path $PSScriptRoot 'Interop.WindowsInstaller.dll'
$msi = Join-Path $PSScriptRoot 'AdvertMSI.msi'

$src = @"
using System;
using System.Reflection;
using System.Runtime.InteropServices;

public class WIWrapper {
    public static object Repro(string path)
    {
        WindowsInstaller.Installer inst = (WindowsInstaller.Installer)Activator.CreateInstance(Type.GetTypeFromProgID("WindowsInstaller.Installer", true));
        WindowsInstaller.Database db = inst.OpenDatabase(path, 0);
        try 
        {
            WindowsInstaller.View v = db.OpenView("SELECT ServiceInstall FROM ``ServiceInstall``");
            return v;
        }
        catch (Exception e)
        {
            string ret = null;
            var innerE = e;
            while (innerE != null)
            {
                ret += ExceptionToString(innerE);
                ret += "----------  InnerException  ----------\n";
                innerE = innerE.InnerException;
            }

            return ret;
        }
    }

    public static string ExceptionToString(Exception e)
    {
        string ret = "Exception FullName: " + e.GetType().FullName + "\n";
        ret += "Assembly Locatoin: " + e.GetType().Assembly.Location + "\n";
        ret += "Exception Message: " + e.Message + "\n";
        return ret;
    }
}
"@

Import-Module $wilib

$type = "WIWrapper" -as [type]
if (-not $type)
{
    Add-Type -TypeDefinition $src -ReferencedAssemblies $wilib
    $type = [WIWrapper]
}

$type::Repro($msi)

[AaronRobinsonMSFT]

@ameltzerMSFT Could you please provide details on how to get Interop.WindowsInstaller.dll?

[AaronRobinsonMSFT]

Actually, do I need a special version of PowerShell to reproduce this issue? How do I get this version of PowerShell? Just the script may not be sufficient for reproduction steps.

[AaronRobinsonMSFT]

I have been able to figure out a local repro.

[AaronRobinsonMSFT]

The problem is at

https://github.com/dotnet/runtime/blob/bd30e154150b8568c684e223af1f623d26bd68b6/src/coreclr/src/System.Private.CoreLib/src/System/RuntimeType.CoreCLR.cs#L4069-L4071

The issue here is that in .NET Framework we didn't pass this flag, but we did catch the TargetInvocationException explicitly and unwrap it to propagate the inner exception. In .NET Core we instead pass the BindingFlags.DoNotWrapExceptions. However we don't respect that flag in this code path - boo.

We have two options:

1) Explicitly check and unwrap the exception similar to .NET Framework.

1) Update the code path to respect the binding flag.

I will investigate (2) and see how much that is going to cost - it really is the right fix.

@ameltzerMSFT I don't know if this will meet the bar for .NET 3.1 servicing. Is .NET 5 okay or do you require this farther back?

Notes for ship room:
(1) is a simple low risk change and I have little concern about applying that in .NET 3.1. Therefore if this fix is going to be added to a servicing change I will apply (1) in .NET 3.1 regardless of how it is addressed in .NET 5.

/cc @jeffschwMSFT

[ameltzerMSFT]

Hi, @AaronRobinsonMSFT. Thank you for investigating this!

We have worked around this in our code, so this issue isn't a blocker for us. (I have a personal opinion that this seems like something worth servicing in .NET 3.1, but I have no insights into your current bug bar or shipping priorities. 😅 )