Runtime: Mono linker failed with function pointer delegates

[hez2010]

Description

Mono linker failed with function pointer delegates.
Code:

#define UNICODE
#define WIN32
#include <cstring>

extern "C" __declspec(dllexport) char* __cdecl InvokeFun(char* (*foo)(int)) {
    return foo(5);
}

using System;
using System.Diagnostics;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

namespace Test
{
    unsafe class Program
    {
        [DllImport("./Test.dll")]
        static extern string InvokeFun(delegate* cdecl<int, IntPtr> fun);

        [UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
        public static IntPtr Foo(int x)
        {
            var str = Enumerable.Repeat("x", x).Aggregate((a, b) => $"{a}{b}");
            return Marshal.StringToHGlobalAnsi(str);
        }
        static void Main(string[] args)
        {
            var callback = (delegate* cdecl<int, nint>)(delegate*<int, nint>)&Foo;
            Console.WriteLine(InvokeFun(callback));
        }
    }
}

Now run dotnet publish with PublishTrimmed:

dotnet publish -c Release -r win-x64 /p:PublishTrimmed=true

Got exception from mono lnker:

  Mono.Linker.LinkerFatalErrorException: ILLink: error IL1005: Test.Program.InvokeFun(method): Error processing method 'Test.Program.InvokeFun(method)' in assembly 'Test.dll'
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Linker.Steps.MarkStep.ProcessInteropMethod(MethodDefinition method)
     at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason)
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue()
     at Mono.Linker.Steps.MarkStep.Process()
     at Mono.Linker.Steps.MarkStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger)

Configuration

.NET 5 preview 8

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmosemsft]

@vitek-karas should this transfer to linker repo?

[EgorBo]

both Android and iOS samples from src/mono/netcore/sample directory fail to build because of this error.

[vitek-karas]

This has been already fixed - see the original linker bug https://github.com/mono/linker/issues/1345.

I just tried the repro from this issue with latest 5.0.100-rc.1.20453.7 and I don't see any errors.
(Note that the syntax for function pointers has changed, now it looks like delegate* unmanaged[Cdecl]<int, IntPtr> fun)

Can you please try with the latest RC build?

[akoeplinger]

For the Android/iOS samples we build with the SDK specified in global.json which is still using a preview8 SDK:
https://github.com/dotnet/runtime/blob/a196d534495a1654ffd158211ca8761b1eba8c05/global.json#L3.

[EgorBo]

@akoeplinger @vitek-karas updating global.json to 5.0.100-rc.1.20453.7 fixed the issue! thanks, I'll update it as part of my pr

[akoeplinger]

@EgorBo we can't do that, it needs to happen as part of the monthly infrastructure rollout (https://github.com/dotnet/runtime/issues/41684)

[marek-safar]

@akoeplinger isn't the rollout scheduled for Monday?

[msftbot[bot]]

Tagging subscribers to this area: @vitek-karas, @agocke
See info in area-owners.md if you want to be subscribed.

[akoeplinger]

yeah but it looks like it's not as simple as bumping the version, see #41684. So I'd let Viktor handle it.

/cc @ViktorHofer

[ViktorHofer]

Thanks for looping me in. Yes, the plan is that I update the SDK as part of September's monthly rollout. Will send out a mail later today about the rollout. Thanks

[ViktorHofer]

@akoeplinger the plan was to update the P8 final. What's the minimum required version that you need?

[akoeplinger]

@ViktorHofer it looks like the error still happens with 5.0.100-preview.8.20359.7 SDK but is fixed in 5.0.100-rc.1.20453.7 based on the comments in https://github.com/mono/linker/issues/1345, so I think a Preview8 SDK is probably not enough.

[akoeplinger]

@EgorBo can you try whether it happens with 5.0.100-preview.8.20417.9 which is the released preview 8 SDK?

[ViktorHofer]

Let's baseline on P8 final (min version) but use an RC1 build as the target version.

[akoeplinger]

I did a quick check with the code from https://github.com/dotnet/runtime/issues/41866#issue-693133270 and it indeed still fails with 5.0.100-preview.8.20417.9.

[msftbot[bot]]

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

[ericstj]

I tested this on latest and it's working.

I needed to update the syntax per: https://github.com/dotnet/csharplang/blob/master/proposals/csharp-9.0/function-pointers.md

Now it is
```c#
using System;
using System.Diagnostics;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

namespace Test
{
unsafe class Program
{
[DllImport("./Test.dll")]
static extern string InvokeFun(delegate* unmanaged[Cdecl] fun);

    [UnmanagedCallersOnly(CallConvs = new[] { typeof(CallConvCdecl) })]
    public static IntPtr Foo(int x)
    {
        var str = Enumerable.Repeat("x", x).Aggregate((a, b) => $"{a}{b}");
        return Marshal.StringToHGlobalAnsi(str);
    }
    static void Main(string[] args)
    {
        var callback = (delegate* unmanaged[Cdecl]<int, nint>)(delegate* managed<int, nint>)&Foo;
        Console.WriteLine(InvokeFun(callback));
    }
}

}

Running on 

dotnet --version
5.0.100-rc.2.20465.15

C:scratchlinkerTest>dotnet.exe publish -c Release -r win-x64 /p:PublishTrimmed=true
Microsoft (R) Build Engine version 16.8.0-preview-20464-02+0220c5eae for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

Determining projects to restore...
All projects are up-to-date for restore.
You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
linkerTest -> C:scratchlinkerTestbinReleasenet5.0win-x64linkerTest.dll
Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
linkerTest -> C:scratchlinkerTestbinReleasenet5.0win-x64publish
```