Runtime: Profiler Interest Group - Announcements

[noahfalk]

I'm making this issue as an informal way to flag other issues and discussions that might be relevant to Building .NET profiling tools. This is similar to the .NET announcements repo, but given that profiling is a small crowd I didn't want to create noise for everyone or be very formal about it. Follow the issue if this is something you care about. Hopefully it works well but if not we can try something else.

Please don't do discussion directly in this issue, just links. Thanks!

[noahfalk]

Retroactive announcement - Oct 20th - I updated the profiling status page that describes our progress supporting the ICorProfiler APIs
PR: dotnet/coreclr#14644
Latest status: https://github.com/noahfalk/coreclr/blob/master/Documentation/project-docs/profiling-api-status.md

[noahfalk]

Retroactive announcement - Oct 20th - @davmason implemented new ICorProfiler APIs (#14612, dotnet/coreclr#14643) to help profilers deal with the breaking changes that are coming with [tiered jitting] (https://github.com/dotnet/coreclr/pull/12193)

[noahfalk]

Jan 4th 2018 - @sywhang brought up more profiling tests for ARM and OSX and we believe things are working well
PR: dotnet/coreclr#15659
Issue: dotnet/runtime#6098
Latest status: https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/profiling-api-status.md

[noahfalk]

Jan 4th - @sywhang fixed a bug in which the profiler invoked AppDomainCreationFinished multiple times for the same AppDomain
Issue dotnet/runtime#6804

[dotnetjt]

Hi @noahfalk - just wondering if the profiling API support for Linux went out in 2.1. I've been coding off and on, still working against a 2.1 preview.

I fired up a new Ubuntu server and installed the 2.1 runtime, and set up environment to point to my (portable) profiler library and it doesn't even attempt to load. I'm pretty sure it's not me as ldd shows that I'm not missing any libraries, so all of my static linking is fine.

Only other thing I can think of is that 2.1 doesn't support it yet.

[noahfalk]

.NET Core 2.1 bug : (
We had an unintended behavior change in the ReJIT profiling API that has caused some profilers to fail. I've got a fix that restores the original behavior in the daily builds and working on getting it into 2.1 servicing releases. If you have a profiler that uses the RequestReJIT API but does not call ICorProfilerFunctionControl::SetILFunctionBody to set the IL you may have been affected. Check out dotnet/coreclr#18448 for more details and please reach out in that issue discussion if you have any questions or concerns I can help with.

[dotnetjt]

Yes, I had seen that. Fortunately, doesn't impact me. :)

[sbomer]

We are adding a startup hook that could potentially be useful for profilers: https://github.com/dotnet/core-setup/pull/4421. The plan is to add a DOTNET_STARTUP_HOOKS environment variable that can be used to inject managed code into the process before Main. If you have comments or suggestions, please leave them in that pull request!

[noahfalk]

.NET Core 2.2 Preview 2 has shipped with Tiered Compilation on by default. We warned this was in progress last year above in this thread:

Retroactive announcement - Oct 20th - @davmason implemented new ICorProfiler APIs (#14612, dotnet/coreclr#14643) to help profilers deal with the breaking changes that are coming with [tiered jitting] (#12193)

If you haven't yet tested with this feature on we hope you will soon : )

[kouvel]

Tiered compilation will be disabled by default for .NET Core 2.2 RTM, mainly due to https://github.com/dotnet/coreclr/issues/19752, which we plan to fix soon for 3.0

[davmason]

dotnet/coreclr#22617 allows profilers to edit more types of metadata after module load (specifically adding TypeDef and MethodDef). However, adding virtual methods after module load is still unsupported.

There is a minor breaking change as part of this work. ICorProfilerInfo7::ApplyMetadata now potentially can trigger a GC, so it is now illegal to call it in situations where a GC is forbidden.

Also included in that PR is a document (located at Documentation/Profiling/Profiler Breaking Changes.md) to track breaking changes as they happen.

[noahfalk]

If you have any questions/concerns about @davmason's changes above, as always feel free to reach out to us. Thanks!

[Maoni0]

the existing COR_PRF_HIGH_BASIC_GC profiling is VERY heavy weight. dotnet/coreclr#22866 added a lightweight GC profiling option that just gives you

• GC start callback
• GC end callback
• update generational bounds

this is something you can use on production machines. you call access this by calling ICorProfilerInfo5::SetEventMask2 and set COR_PRF_HIGH_BASIC_GC in dwEventsHigh.

[davmason]

There are three recent changes to announce.

First, profiler attach/detach is now supported (#24670) leveraging the existing EventPipe work. You can find documentation on how to use it here: https://github.com/dotnet/coreclr/blob/master/Documentation/Profiling/Profiler%20Attach%20on%20CoreCLR.md. This means that the trigger process will use a completely different API, but the advantage is that it works on Linux and Windows.

Second, a new api ICorProfilerInfo10::RequestReJITWithInliners has been added (#24461) to allow profilers to ReJIT after attach. Documentation for ReJIT on attach is here: https://github.com/dotnet/coreclr/blob/master/Documentation/Profiling/ReJIT%20on%20Attach.md.

The last announcement is that ICorProfilerInfo2::DoStackSnapshot is now supported on Linux (#24968). On Linux you can call DoStackSnapshot from the same thread you are running on at any time, if you would like to get a snapshot of other threads there is now ICorProfilerInfo10::SuspendRuntime to stop all managed threads (without performing a GC) and make them safe to snapshot, and then ICorProfilerInfo10::ResumeRuntime to resume the runtime once you are finished collecting snapshots. It is not possible to snapshot an arbitrary thread on Linux. Windows support remains unchanged.

Please feel free to reach out with any questions, comments, or concerns.

[davmason]

Hi Everyone,

I recently opened dotnet/coreclr#26762 to change how we deal with an attach profiler on shutdown. Currently on graceful process exit (i.e. the C# main exits normally) we issue a ICorProfilerCallback::Shutdown call and then release the profiler's COM object we hold on to, but we skip the callback and release on non-graceful shutdown.

This PR will change it so the runtime never releases the profiler's COM object, and instead lets the OS clean up during process shutdown. You will still get a ICorProfilerCallback::Shutdown call if the process is terminating gracefully.

It is my belief that this shouldn't have a negative impact on profilers, since you already have to account for the situation where the runtime doesn't free your library. Now that situation just gets a lot more common.

If you would be impacted by this change please get in touch. This issue is just for announcements so please make any comments either in the PR or on the relevant issue dotnet/coreclr#26687.

[sawilde]

@davmason are you saying that you will always do an AddRef but never do a Release?

[davmason]

@davmason are you saying that you will always do an AddRef but never do a Release?

Yes, that is the new behavior.

[davmason]

Hi Everyone,

PR #1201 means that arrays are no longer special cased on class unload and profilers will be notified (via ClassUnloadStarted) when an array type is unloaded from a collectible context. Up until this point they have been purposefully excluded from ClassUnloadStarted.

I don't expect this to be a breaking change for any profilers, please get in touch if this is not the case.

[NitzanEgozy]

Hi Everyone,

PR #1201 means that arrays are no longer special cased on class unload and profilers will be notified (via ClassUnloadStarted) when an array type is unloaded from a collectible context. Up until this point they have been purposefully excluded from ClassUnloadStarted.

I don't expect this to be a breaking change for any profilers, please get in touch if this is not the case.

Thanks @davmason,
Currently (ever since CLRv4, including CoreCLR), neither ClassLoad* or ClassUnload* callbacks are called for arrays. Will this change ClassLoad* callbacks too?

[davmason]

Currently (ever since CLRv4, including CoreCLR), neither ClassLoad* or ClassUnload* callbacks are called for arrays. Will this change ClassLoad* callbacks too?

As it stands now the class load callbacks are not called. That is because this wasn't an intentional change, but rather a side effect of removing array's TypeDesc.

Is that an issue for your profiler? It would be a fairly small change to have array types reported for class load, or to block them from being reported in class unload and go back to the old behavior.

[NitzanEgozy]

Currently (ever since CLRv4, including CoreCLR), neither ClassLoad* or ClassUnload* callbacks are called for arrays. Will this change ClassLoad* callbacks too?

As it stands now the class load callbacks are not called. That is because this wasn't an intentional change, but rather a side effect of removing array's TypeDesc.

Is that an issue for your profiler? It would be a fairly small change to have array types reported for class load, or to block them from being reported in class unload and go back to the old behavior.

No, this shouldn't be an issue (as we're supporting CLRv2/4 anyway). Thanks for these updates.

[davmason]

Hi again everyone,

I would like to give you all a heads up about a potential breaking change.

32283 introduced a new GC generation for the first time since very early in desktop .net. This means that there are now 5 total generations instead of the previous 4:

• Gen 0
• Gen 1
• Gen 2
• Large Object Heap
• Pinned Object Heap

There is no specific breaking change to the ICorProfiler APIs, but I suspect that many of you will be broken because of assumptions made about the number of GC generations inside your profilers. We (the .net diagnostics team) are finding many of our tools need updating because of this change. If you are interested in the type of work we need to do, you can find that at #35561.