Runtime: Allow ref struct to contain ref fields

[Sergio0694]

The current plan is to add ref fields as first class construct to the type system. See proposal at https://github.com/dotnet/csharplang/pull/3936 for more details.

---

Original proposal

Overview

Make the System.ByReference<T> type (here) public instead of internal.

Rationale

It's already possible to achieve the same behavior of ByReference<T>, namely to store a ref T to some variable as a field in a ref struct, by leveraging the Span<T> and MemoryMarshal types, like so:

public readonly ref struct FakeByReference<T>
{
    private readonly Span<T> span;

    public ByReference(ref T value)
    {
        span = MemoryMarshal.CreateSpan(ref value, 1);
    }

    public ref T Value => ref MemoryMarshal.GetReference(span);
}

This can either be used as is, as a field in some other ref struct type, or the same technique (storing a 1-length Span<T> in place of the missing ByReference<T> type) can be used in other ref struct types to reproduce this same behavior: storing a managed reference as a field in a ref struct type.

Given that this is already possible with existing APIs, as shown above, but it's less performant than the original ByReference<T>, more verbose and more error prone, why not just make the original ByReference<T> type public so everyone can use it with no need to reinvent the wheel?

It'd be a neat API to have, and at basically no cost, since it's already in the .NET Core BCL anyway. 😄

[jkotas]

We would prefer this to be exposed as ref fields in C# language: https://github.com/dotnet/csharplang/issues/1147 . This would be even faster, leaner, and less error prone than ByReference<T>.

[Sergio0694]

Hi @jkotas - thanks for linking that issue, I had missed that! 😊

Is there an ETA for that though? By the looks of it, and the fact it has been opened for over two years already, it looks like it still has a way to go, plus all the additional difficulties caused by the fact that that would require at least some Roslyn updates to support that, if not some CLR changes too.

My rationale here was that making the ByReference<T> type public wouldn't allow new scenarios not achievable today, but simply make them more convenient for devs that were interested in using them (with the assumption that those devs would already be aware of all the various implications of using unsafe APIs in that case). Even if a ref T field feature was to be implemented one day, wouldn't this still be a nice and easy to implement stopgap for the time being? Again, given that the same exact feature can still be achieved today anyway, just less efficiently and with more room for mistakes (and the additional verbosity of having to reimplement that fake ByReference<T> type).

[jkotas]

We are here for the long haul and want to do things properly. We try hard to avoid exposing temporary workarounds that we are stuck forever even once the more appropriate solution is available.

[Sergio0694]

Sure thing, that's fair enough 😄
Also, just to be clear, it wasn't my intention to suggest this change as a quick and sloppy solution instead of doing things properly, as I said my rationale was more of an "iterative" improvement for a scenario that's technically supported already, but with an even worse implementation. But I can see how it'd be preferable to jump straight to the optimal and built-in solution, absolutely 🚀

Is there any approximate ETA on that proposal? I see the issue has been open for over two years already and it's not currently assigned to any milestone, not even X.0 or X.X. Is that a feature that's still being evaluated and not yet decided on, or is it something the various teams have already decided to implement, just not with an exact timeframe for it?

Thanks again for your time! 😊

[jkotas]

It did not make it into the list of C# features we are looking at for .NET 5.

[DaZombieKiller]

Wouldn't exposing this type make it possible to create ref fields in languages other than C#?
To me that sounds like something worth considering.

[jkotas]

This type as it exists today is not safe construct. It is very easy to create dangling pointers with it, without guardrails in the language. If we were to expose it, we would likely mark it with the special ObsoleteAttribute to block its use in languages that are not aware of it (e.g. similar to how ref structs are blocked).

[GrabYourPitchforks]

On a related note, Jan, you and I previously talked about a syntax that would allow operating with gcrefs as naturally as operating with pointers. Things that would allow natural syntax like the + operator without bouncing through Unsafe.Add. Would something like that ever make sense to add as a language feature, or as a ByRef-like type, or as something else?

[Sergio0694]

Hey @jkotas - I have a follow up question on this, specifically regarding the general criteria being used to expose APIs or add explicit support for "unsafe" operations. You mentioned how with the ByReference<T> type is't possible to create dangling pointers, which is certainly true. My question though is about the seemingly arbitrary decision not to make the type public, despite the BCL having a ton of potentially dangerous APIs already. Consider this snippet:

public static Span<T> GetFaultySpan<T>()
{
    T value = default;
    return MemoryMarshal.CreateSpan(ref value, 1);
}

This compiles fine, but results in a dangling pointer just like the one you mentioned. It's not perfectly clear to me why is this API public, but ByReference<T> is not. Don't get me wrong, this API is super useful (and I love the fact that it's available!), but I don't get why the same rationale doesn't allow to ByReference<T> as well: "here's a potentially dangerous but useful tool, use at your own risk!".

Or, just like the BCL already includes dedicated code to properly handle devs doing stuff that's technically not valid (eg. Memory<T> wrapping strings as mutable), then why not just use the same rationale here as well, and provide a tool that enables devs to do exactly the same thing they can already do with MemoryMarshal.CreateSpan, but just in a more efficient and less error prone way?

I just can't stress enough how useful the ByReference<T> type would be, and with the native support for ref T fields not making the cut for .NET 5 I'd say it'll be at least another year (if not more) until that eventually makes it to a public release? As you said, you could also mark it as [Obsolete] just like Span<T>, to avoid issues with other .NET languages.

Just though I'd share my thoughts on this, thank you in advance for your time! 😊

P.S. Just so I'm perfectly clear, I'm not criticizing your work in any way, the .NET team is doing an incredible job and I really love where the whole platform is going! Just trying to make my case here as to why I think making ByReference<T> public could be a win-win for everyone. 😄

[tannergooding]

@jaredpar had some reasons on why this wasn't safe from the language perspective and would be extremely likely to cause bugs without proper language support for tracking the lifetime.

[jkotas]

This compiles fine, but results in a dangling pointer just like the one you mentioned

Methods in System.Runtime.InteropServices are unsafe. They can produce dangling pointers, corrupt memory, etc. You have to know what you are doing when using them and they should be avoided where possible.

extremely likely to cause bugs without proper language support for tracking the lifetime.

Correct. We would like ref fields to have proper language that guarantees its safety.

[Sergio0694]

Ah, yeah that's a fair point, having ByReference<T> under System.* would definitely be a problem - I hadn't noticed that before and I had just assumed it was under System.Runtime.* somewhere like Unsafe or MemoryMarshal. I guess it could very well be moved there since right now it's internal, but I can see how that would require more work/refactoring than just making it public 👍

And yeah I agree that just having language-level support for ref fields would be even better, absolutely no dispute there. I was just hoping for a better workaround for the time being, seeing as that feature at least for now seems to be pretty far from completion, is all.

Thank you both again for your time, I really appreciate the conversation here!

[jaredpar]

@tannergooding

had some reasons on why this wasn't safe from the language perspective and would be extremely likely to cause bugs without proper language support for tracking the lifetime.

The reasons are pretty straight forward: the lifetime safety rules, as written today, are predicated on such a constructor not existing. It's explicitly called out in the design.

That's not to say we can't do ref fields, just that the current design was written explicitly assuming they didn't exist. The initial designs of ref struct actually had support for them but we removed them because it introduced some heavy complexity into our lifetime rules. Now we think we have a simpler way of doing this and hope to bring ref fields into a future version of C#.

[john-h-k]

+1 on this. While the span workaround, well, works, it isn't ideal. Particularly where you have an interop scenario where you are using ByReference<T> in place of a T* in the native code, and want it to be blittable so it can be passed when pinned

[john-h-k]

On a related note, Jan, you and I previously talked about a syntax that would allow operating with gcrefs as naturally as operating with pointers. Things that would allow natural syntax like the + operator without bouncing through Unsafe.Add. Would something like that ever make sense to add as a language feature, or as a ByRef-like type, or as something else?

Having it actually added to refs would be a huge breaking change wouldn't it. As currently, + refers to the underlying object the ref points to

[Sergio0694]

Personally I'd already be more than happy just to have the ref field C# feature that was mentioned, especially if not having to also work out a new syntax to allow offsetting over ref-s could make the implementation of these ref fields quicker. Even just with that, we could write something like:

public readonly ref struct Ref<T>
{
    public readonly ref T Value;

    public Ref(ref T value)
    {
        Value = value;
    }

    public ref T this[int i]
    {
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        get => Unsafe.Add(ref Value, i);
    }
}

Which personally I'm very looking forward to, as it'd be a nice and fast abstraction over having to manually use Unsafe.Add every time you need to offset to some location 😄

[john-h-k]

Personally I'd already be more than happy just to have the ref field C# feature that was mentioned,

well yeah, that is just better, but it is far away, will take a lot of compiler work, new language rules, etc. i just want the dangerous scary thing exposed that if you f#@! up then it doesn't help you

[GrabYourPitchforks]

~~If you _reeeeeeeally_ wanted ByReference<T>, an enterprising individual could write a fully out-of-band implementation. 😁 You'd have to write it in IL instead of C#, but there's nothing stopping anybody from doing it. Said assembly would need to target netcoreapp2.1+ and wouldn't be compatible with netstandard or netfx.

I don't think such an out-of-band type would be blittable or would play well with the interop system. I don't even think the internal in-box type fulfills these characteristics. You'd need to pin the _T&_ to a _T*_ if you wanted it to be blittable.~~

This idea is shot. See below.

[jkotas]

@GrabYourPitchforks I am not sure what you have in mind, but I do not think it would work.

[john-h-k]

@GrabYourPitchforks ByReference<T> is not an S.R.CS.Unsafe kind of type which is written in IL. It is an intrinsic'y one that has a special meaning to the runtime afaik

[GrabYourPitchforks]

@jkotas I thought as of netcoreapp2.1 the runtime itself had support for byref members of structs, even though the C# compiler doesn't? I was thinking you could write a class entirely in IL that looked something like:

// pseudo-code
public ref struct ByReference<T>
{
    private readonly T& _value;
    public ByReference(ref T value) { _value = ref value; }
    public ref T Value => ldfld(_value);
}

The public API surface uses only constructs that the C# compiler can handle. Admittedly I haven't actually _tried_ this. Just spitballing.

[jkotas]

I thought as of netcoreapp2.1 the runtime itself had support for byref members of structs

That is not correct. You should see type load exception if you try to run your example.

[john-h-k]

@GrabYourPitchforks nope, invalid IL. I believe the current way it works is by having a dummy IntPtr in the ByReference<T> type (see here), and specially instructing the runtime that in reality is a tracked ref

Won't even assemble, iirc, @jkotas - FieldSig has no BYREF attribute (unlike LocalVarSig/Param blob) so it can't be encoded.

[GrabYourPitchforks]

Well, there goes that idea. :)

[ENikS]

@jkotas

I been scratching my head for for a very long while trying to create a struct that holds reference to a parent.

I am using it in Unity container to build dependency graphs during resolution. I had to use unsafe code to do so but it places certain limitations on where it could be executed. A solution that doesn’t require unsafe methods would be a dream come through for the project but every time I see potential fix it is either internal, or does not compile.

Perhaps you could recommend something that I missed?

[jkotas]

You have to use unsafe code for that today.

[ENikS]

@jkotas

Any way I could build my own ByReference<>? If I can build entire framework locally, why not just that single type?

NET is full of dangerous code and could be used badly, I am baffled why that particular type became such a guarded 'golden fleece'? For library authors it could be life saver. The closest thing C# could have to C++.

Write "Don't use under no circumstances!" in big letters in documentation and that will scare away 99% of developers, just these who have no other choice will dare to proceed...

[DaZombieKiller]

Any way I could build my own ByReference<>? If I can build entire framework locally, why not just that single type?

There is an example of this in the OP, using a private Span<T> field.

[ENikS]

@DaZombieKiller Thank you, but it is rather expensive to create 3 different types (ByReference, Span, FakeByReference) to hold just one reference.

In the IoC I am working on even difference in one reference field in the context structure brings performance down 15% from 14ns to 16.5ns. Multiplied by couple of million resolutions during startup of application adds up a lot.

[jkotas]

Any way I could build my own ByReference<>?

I would just use unmanaged pointers. Without the language safety guardrails, ByReference<T> and unmanaged pointers are equally unsafe.

[Sergio0694]

@ENikS I've added a Ref<T> type (and readonly version, and Nullable ones too) to the Microsoft.Toolkit.HighPerformance package, if you're interested. You can find the docs for that here. I also have a PR open (should be merged shortly) that makes it use the actual ByReference<T> type from CoreCLR when running on .NET Core, so you also get exactly the same codegen/performance as using ref T fields in that case (if they actually existed). I find that type does the job quite well working around the current lack of support for this feature in C#.

[ENikS]

@jkotas I can't use pointer because structs contain managed references.

To give you more 'context', when IoC advances from dependency to dependency it creates a local Context on the stack frame. But unfortunately it also require a reference to parent Context living on stack frame as well. The lifetime is guaranteed to be shorter than the parent because parent is always calling method and survives longer.

Implementing this in unsafe code places restrictions on execution in certain environments. I have to use class instance (I have a class and a struct with the same signature) in place of struct to mitigate but doing so degrades performance due to heap allocations. As you can see, I would sell my next unborn child to have fast native support for ByReference

@Sergio0694

Thank you, I will check it out!

[jkotas]

I can't use pointer because structs contain managed references.

You can use System.Runtime.CompilerServices.Unsafe to get around this limitation.

[DaZombieKiller]

That pull request (https://github.com/windows-toolkit/WindowsCommunityToolkit/pull/3367) has me not just impressed but concerned, as it's exposing a type that is (to my knowledge) considered by the .NET team a temporary workaround and may be removed when a more appropriate ref field solution comes along. If several projects start relying on it, then wouldn't that place the removal of ByReference<T> in a bit of a tough spot? Or is this going to be considered along the same lines as reflection (you choose to use it, you choose to accept breakage) for now?

[ENikS]

@jkotas

You can use System.Runtime.CompilerServices.Unsafe to get around this limitation.

I can't! It will not run in some of UWP scenarios. Also, Unity is used by a lot of big financial organizations and some place restrictions on Unsafe code in libraries.

[jkotas]

It will not run in some of UWP scenarios

Then you need a slower implementation for those. Even if the proper support for byref fields gets added to the language and runtime, this support won't be backported to existing runtimes.

Unity is used by a lot of big financial organizations and some place restrictions on Unsafe code in libraries.

There is no type and memory safe way to do byref-fields today. If your requirement is to use type and memory safe code only, you cannot use byref-fields today.

[jkotas]

is this going to be considered along the same lines as reflection (you choose to use it, you choose to accept breakage)

Yes. https://github.com/windows-toolkit/WindowsCommunityToolkit/pull/3367#issuecomment-676557220

[ENikS]

@jkotas You are not helping!

The decision seems to be rather arbitrary as there are workarounds that could provide same behavior but with higher costs. Keeping that type internal does not provide any additional measure of safety, it just prevents proper optimization.

Would it be possible to bring in more decision makers into this discussion to weigh pros and cons?
Since net5 release is still some time in the future, we have time to properly discuss it...

I've read a lot of @VSadov comments related to this and other issues, perhaps he can provide any suggestions?

Lets start with a short list of people who could make this decision. @jkotas do you know who are these people, could you provide their names?

[jkotas]

You can find list of area owners at https://github.com/dotnet/runtime/blob/master/docs/area-owners.md

net5 is feature complete. We are switching master branch to net 6. The changes going into net 5 release branch at this point are fit and finish bug fixes only.

[ENikS]

In my view this type being internal is a bug :)

[Sergio0694]

@ENikS About UWP, if you're tracking references to fields within objects on the heap, you can use the same pattern that is used by both the portable Span<T> type and the Ref<T> type from the HighPerformance package when running on runtimes without runtime support for Span<T> (ie. without the internal ByReference<T> magic type): just have your type contain an object field to the parent, and an IntPtr offset for that field. Then you can get a ref T to that field by just shifting the reference.

You can use the object.DangerousGetObjectDataByteOffset<T>(object) and object.DangerousGetObjectDataReferenceAt<T>(object, IntPtr) extensions (still from that HighPerformance package) to do that. They'll take care of all the weird offsetting for you.

This approach works on even .NET Standard 1.4 (so also on eg. Mono, etc.), it's mostly just slower and it can only work for interior references to objects, so it's not quite as flexible. It does the job though and it's still pretty useful in some cases 😊

[ENikS]

@Sergio0694 Excellent suggestion, thank you

[Sergio0694]

@ENikS FYI that's actually exactly how Ref<T> works too when running on UWP (or .NET Standard <= 2.0 in general).
It offers a Ref<T>(object, ref T) constructor that does exactly that behind the scenes.
Just in case you don't want to reimplement that from scratch (or, you can look at the source as a reference).

[GrabYourPitchforks]

The decision seems to be rather arbitrary as there are workarounds that could provide same behavior but with higher costs. Keeping that type internal does not provide any additional measure of safety, it just prevents proper optimization.

I don't think it's arbitrary. https://github.com/dotnet/runtime/issues/32060#issuecomment-596624746 gave some examples of where the compiler's safety rails fall apart were this type to be publicly exposed.

The comments in this thread from the runtime and libraries and language team are all pretty consistent that we'd like to expose this feature the correct way - via proper ref fields - rather than exposing the internal ByRef<T> type. Once that change comes online, the ByRef<T> type will probably be deleted from the runtime. (This also means that the Windows Toolkit PR mentioned earlier will stop working on this new runtime.)

[Sergio0694]

@GrabYourPitchforks I mentioned future support plans in my comment here, that's one of the reasons why we didn't just expose ByReference<T> directly, but still abstracted it through our Ref<T> types (and related types), to give us more flexibility.

If with eg. .NET 6 (or some other future runtime) the ByReference<T> was removed and proper support for ref T fields was added, we'd just tweak the code accordingly - the HighPerformance package is already multi-targeting a number of different runtimes/profiles specifically to be able to deal with runtime-specific quirks too when needed (eg. we offer fast indexers for arrays on .NET Core that do the unsafe cast through an internal RawArrayData type mapping the array layout on CoreCLR) 😊

[VSadov]

It is a pervasive assumption in GC that byref fields do not exist. That is not going to change. A lot of things are substantially simpler/faster due to this assumption.

ByReference<T> gets around this limitation by colluding with JIT - formally there is no ref field, but JIT reports roots as if there is one.

Basically this type works as expected only with a JIT/AOT compiler that knows about it and only in a few scenarios where JIT has a chance to do its thing. That is enough to implement span types.

Supporting this type more generally will at least require a lot of testing on a range of platforms and commitment to keep that working in the future.

[ENikS]

@GrabYourPitchforks

The protection provided by keeping ByReference internal is easily circumvented as shown in OP. But instead of doing it properly, it is done as a slow hack. In a way it is like an abortion, when outlawed, it is still done, but instead of going to a doctor, it is done by charlatans (no relation to anyone :).

@VSadov
The type is being used left and right in framework, I don't know how much more testing should be done.

With a push to use value tapes in .NET it is becoming real impedance when it could not be referenced

[jkotas]

The type is being used left and right in framework,

ByReference<T> is as an implementation detail in 3 low level types in CoreLib (Span, ReadOnlySpan, TypeReference). I would not call it used left and right.

[ENikS]

Considering that Span is being added to each and every high performance API (http, stream, buffer, pipeline, and etc.) I would say it is.

[GrabYourPitchforks]

The protection provided by keeping ByReference internal is easily circumvented as shown in OP.

Keeping things internal isn't meant to be a "protection" _per se_, so there's nothing to circumvent. It's internal because per the runtime team (of whom many have commented on this thread), we don't feel it's a good type to expose. When we expose a type publicly, at that point we sit down and hash out the full design, including any necessary language support or how the type is exposed by our API surface.

There's lots of other internal functionality that can be accessed via things like private reflection. For the most part we don't go out of our way to block callers from using them - doing that would just lead to an arms race. But we also don't support these scenarios at all, and apps who do this need to be prepared for things for things to break unexpectedly. It would be perfectly valid for the implementation of _SomeInternalFrameworkMethod_ to change from "please multiply two numbers together" on Tuesday to "please reformat my hard drive and delete all my cloud backups" on Wednesday.

The type is being used left and right in framework ... Considering that Span is being added to each and every high performance API (http, stream, buffer, pipeline, and etc.) I would say it is.

I think you've drawn an incorrect conclusion here. You're correct that span is a fundamental building block of high-performance APIs, but I think it's a leap to say that span's internal implementation details are fundamental building blocks. The point of an implementation detail is that we can rip it out and replace it in the future without affecting any callers or consumers. Once ref fields come online and we rewrite spans to use that feature instead of ByRef<T>, then _poof!_ suddenly ByRef<T> has exactly zero direct or indirect consumers. Clearly an implementation detail that can be wholesale replaced without affecting any consumers is not a fundamental building block.

[ENikS]

It's internal because per the runtime team (of whom many have commented on this thread), we don't feel it's a good type to expose. When we expose a type publicly, at that point we sit down and hash out the full design, including any necessary language support or how the type is exposed by our API surface.

Perhaps this issue could be revisited and based on feedback from developers the team could readdress this decision? Times change and requirements change as well. Not so many cared about struct types except to marshal across native boundaries.

Now it is everywhere and inability to properly reference these impedes development for many, as you could see in this and other referenced threads.

I understand there is a proper review procedure and decisions are not made lightly. I just want these who make these decision to address it here and explain where it is going and how soon. I want to start formal process to get this moving.

I would be equally happy with Yes and No. If it is No, I'll just use my own solution and move on, but I would rather prefer Yes.

Once ref fields come online and we rewrite spans to use that feature instead of ByRef, then poof! suddenly ByRef has exactly zero direct or indirect consumers. Clearly an implementation detail that can be wholesale replaced without affecting any consumers is not a fundamental building block.

I've seen no mention of these in either C# 9, nor net 5.0, so I would not hold my breath.

[ENikS]

Please consider this as well: inability to use this (or similar) API creates unfair advantage to MS code on certain platforms (UWP for example)!

Without ability to effectively reference value types third party code can not compete in speed with APIs released by Microsoft. In comparison, any code, which is part of the MS platform, could use it freely without violating any security policies associated with Unsafe code.

[DaZombieKiller]

Please consider this as well: inability to use this (or similar) API creates unfair advantage to MS code on certain platforms (UWP for example)!

Without ability to effectively reference value types third party code can not compete in speed with APIs released by Microsoft. In comparison, any code, which is part of the MS platform, could use it freely without violating any security policies associated with Unsafe code.

This... is a bit of a stretch. There's nothing stopping you from using ByReference<T>, as seen in the aforementioned pull request. It's merely not officially supported or recommended, especially because there are plans to offer a _proper_ solution for this sometime in the future.

[Sergio0694]

Yeah I agree with this - I had originally opened this issue well before adding the current Ref<T> type to the HighPerformance package, and was curious to hear the official plans for a feature like this. The reasons that Jen, Levi, Stephen and others have mentioned for not making ByReference<T> public as is are perfectly valid indeed, and it does make sense to keep the type internal. Making it public would mean having either having just another tricky to use API to support (I mean just like MemoryMarshal.CreateSpan) or it would require time to offer proper support for it to avoid it possibly being used incorrectly, and at that point it does make sense to just wait and eventually offer a built in ref T field support in C# directly 👍

@ENikS if you really care about having this feature right now, you can just use that type from the Toolkit, and once that PR gets merged you'd get identical codegen too. Just really make sure to read the docs and to understand when/how it can be used correctly, that is, either to:

• Wrap a raw pointer to unmanaged memory
• Wrap an interior reference to an object on the heap (eg. an array item)
• Wrap a local variable only if you use it in stack frames below and never bubble it up.

[GrabYourPitchforks]

I wonder if this is indicative of a deeper communication problem? There's definitely interest from the runtime side in doing this correctly (namely, via ref fields), which would obviate the need for any public ByRef<T> struct. As a strawman syntax:

public readonly ref struct ReadOnlySpan<T>
{
    private ref T _pointer; // ref fields could only be defined in ref structs
    private int _length;
}

And ideally non-runtime code would be able to do the same:

public ref struct MyRefStruct
{
    public ref int _someRefField;
    public int _someNonRefField;
}

This isn't trivial work. It would require changes to the type loader, the VM / JIT, C#, and possibly other mechanisms as well. So as with all things, as it's weighed against other pending work items it could float above or fall below the cut line.

I _think_ we're properly messaging this, but I'm also biased because as a member of the runtime team I see the entirety of the decision-making process. The process is not a black box to me. If in reality we're not communicating this properly, or if it seems decisions are being made arbitrarily, that would be good feedback for us. Please let us know if this is the case.

[ENikS]

I am struggling to implement basic things just because this one type is not public. I do understand your desire to do it properly but it does not help me now! Word struggling is not an exaggeration here, it is real thing for me.

I am replacing types, where appropriate with structures to provide better performance, but UWP and business reasons preclude me from using Unsafe code freely. Having heavy historical participation of Financial institutions, I cannot just ignore their needs.

The solution for this issue exists since 2018, but it is still internal after 2 years of pleas from developers.

There used to be Nullable T, now we have language support for it, but it newer went away. Same with ByReferene, no matter how you call it, it is just a glorified IntPtr

[GrabYourPitchforks]

I am struggling to implement basic things just because this one type is not public.

This is interesting to us! Please elaborate on this point.

One of the things I try to do is to separate "the means" vs. "the ends". Sometimes I'll have a particular scenario ("the ends") in mind, come up with a way ("the means") of accomplishing the task, and become so intensely fixated on lobbying for the mechanism I came up with that I lose sight of the fact that the scenario itself is what's truly important.

A recent example from my own experience is https://github.com/dotnet/runtime/issues/27935#issuecomment-582221872. I was vociferously arguing that the API should have a specific format (returning Range), but the rest of the team brought the conversation back to "What are you trying to do? Can we accomplish your scenario _and_ make the API more approachable if we follow this other style instead?" And the answer was yes. Maybe not quite as cleanly as I had originally wanted, but it nevertheless worked, and the chosen solution is probably better for the ecosystem long-term than the solution I had originally proposed.

But enough about me. 😀
If you're struggling to accomplish a given scenario, I'd strongly recommend opening an issue tracking that scenario specifically. That way people can recommend interim solutions that you can immediately enact. And a proper long-term solution can be tracked.

I am replacing types, where appropriate with structures to provide better performance, but UWP and business reasons preclude me from using Unsafe code freely. Having heavy historical participation of Financial institutions, I cannot just ignore their needs.

You might find the discussion at https://github.com/windows-toolkit/WindowsCommunityToolkit/pull/3367 interesting. Part of that discussion focuses on balancing the risk of calling into the runtime in an unsupported fashion vs. the perf gains of doing so. The maintainers of the HighPerformance library will make the assessment they feel is appropriate for their library. If your clients include financial institutions and other risk-averse parties, you should ensure that the HighPerformance library's assessment is in line with the risk assessment your clients expect of your work.

[ENikS]

@Sergio0694 I missed your original point and kept at it even after getting what I needed, my bad, excellent work on the PR!
@GrabYourPitchforks It is exactly what I was looking for! Many thanks!

As for the struggle, this is very common scenario:

• I need to create a Context struct with Parent being another Context struct allocated within calling method and so on. Basically each method allocates it’s own Context with reference to Paren. Parent is passed to a method by reference method(ref Context c) The struct has to implement interface (long story...) so it is not a ref struct, although it only exists on stack.

• second case is implementing ValueActivity with the same parent/child problem for struct living on the stack

To prove my point that without ByReference runtime is incomplete and should be available to general public, I would like to challenge runtime team to come up with a solution for this problem without resorting to unsafe code.

[ltrzesniewski]

The struct has to implement interface (long story...) so it is not a ref struct, although it only exists on stack.

Even though you could force a ByReference<T> field into your struct the same way the PR does (you'd need to remove the ref in ByReference<T> in the shim assembly), I'm pretty sure the runtime would refuse to load a type constructed like this.

The runtime already lets you do what you need by using pointer fields in your structs (as stack data is not moved around by the GC). If your employer prevents you from using unsafe code, well... I wouldn't say that's a problem in the runtime. 😉

Anyway, I agree that adding "real" ref fields would be a better solution than making ByReference<T> public, as once you make a type public, you need to support it forever.

[Sergio0694]

I agree with both Levi and Lucas here, this specific problem seems more about the actual restriction against unsafe code that @ENikS seemingly has in the project rather than an actual missing feature. Even if proper ref T fields were available (or just a public ByReference<T> type), they would've still not worked in this case due to the use of a non-ref type 😕

@ENikS you should've mentioned you were dealing with a non-ref struct earlier, we would've all told you that if that's the case then neither the actual ByReference<T> type nor the alternative in the toolkit would work (or even compile at all) for you.

I think you should just convince your team/manager/whoever to remove the unsafe restriction and let you use that, especially considering that that restriction is basically 100% meaningless today, as with the Unsafe class (and others) you can still do pretty unsafe things and completely trash the type safety in the runtime (in a way that class is even more unsafe than having pointers).

If you managed that, provided you took extra care to always avoid boxing, the solution with pointers would be perfectly fine, and you'd also get bonus points for faster performance compared to using GC refs (as with raw pointers the GC doesn't even need to update its internal metadata to track those references, which indirectly removes overhead). Second bonus points, using raw pointers would mean the code would work perfectly fine even on UWP or other runtimes targeting .NET Standard <= 2.0, where there isn't even runtime support for spans (so they don't have the internal ByReference<T> type at all). Again, this would both be consistent across runtimes and work downlevel, it'd also be slightly faster, and it's also really your only option anyway, since your type implements an interface and therefore could've never had a ref struct field anyway.

@GrabYourPitchforks I really appreciate the conversation here, and (speaking for myself here) I think you guys are coming across perfectly fine on this! You've all took the time to explain the reasoning not to expose this type (which again make perfect sense and I can agree with them) and you've also provided code snippets to help explain why having such a type being public would be particularly dangerous to use for users and hence not be a good fit for a public type in the runtime. So personally I've also learnt quite a bit on the topic too while reading comments here and in the other related issue (https://github.com/dotnet/csharplang/issues/1147) 😊

Side note, if it helps clear up further confusion, we could also close this issue to make it clearer for others that this actual proposal is not really active anymore, and this issue was mostly just about the conversation on future ref T fields support at this point.

[ENikS]

@ltrzesniewski @Sergio0694
Restriction on Unsafe code comes from UWP environment! It will not execute an App store application with Unsafe code.

[ENikS]

@Sergio0694 could you please postpone closing this issue until runtime team had a chance to suggest a way to solve paren/child challenge for structures? This is not a trick challenge, I am trying to solve it in a real production project.

I am hoping to get an advice and a working example that can run in UWP, In my mind it can only be done through ByReference but perhaps someone in the team knows better way?

@jkotas ?

[Sergio0694]

@ENikS I can guarantee you that that part about UWP restricting unsafe code is 100% incorrect. I have multiple UWP apps published in the Microsoft Store, all using both unsafe code and lots of Unsafe/MemoryMarshal etc. - that's absolutely not a problem. The certification doesn't check that at all, and .NET Native works perfectly fine with that, as it's all completely valid IL code anyway.

Maybe you're just confusing the fact that UWP (I mean .NET Native) only exposes APIs corresponding to .NET Standard 2.0, so it doesn't "officially" have support for the fast Span<T> types? Because that part is correct, yes (though it actually does have the fast span behind the scenes).

[ENikS]

I can guarantee you that that part about UWP restricting unsafe code is 100% incorrect

Are you accusing me of lying? Don't take my word for it, here is an issue to prove it: TypeAccessException in limited sandbox. Limited sandbox is pretty common scenario in high security systems in finance world, as apparently is in Office 365 too, I had other people complain about this problem, so it is not one off case.

[Sergio0694]

@ENikS That was absolutely not my intention, I'm sorry if it came across that way. I've been trying to help out here since you posted that first question two days ago - what I meant in my previous message was _not_ that you were lying (what would even be the point?), but that you might have been confusing your issue with something else, is all.

My point is, I know for a fact that both unsafe and Unsafe code _does_ work perfectly fine in UWP, so your statement that, and I quote, "[...] the UWP environment will not run a Store application with Unsafe code" is simply not accurate. The issue you're seeing will surely be some combination of one of more other (potentially related) factors - here I was just clarifying that unsafe code per se does run fine on UWP and is perfectly supported.

I can agree that UWP does have a lot of limitations and constraints, I was just clarifying that unsafe code on its own is not one of them though 😊

[GrabYourPitchforks]

Re: sandboxes, code access security, and visibility checks - these are no longer supported as security boundaries by the product team. We don't test new language features like ref returns and stackalloc against these environments, nor do we test libraries like System.Memory against these environments. Not a single line of code produced by this team (including any theoretical public ByRef<T> type) is guaranteed to work in such an environment.

[ENikS]

@GrabYourPitchforks

I think we are running in circles and prioritizing discussion on why it is not public instead of what could be done to make it available.

Everyone understands the need for the feature. The team insists on doing it right (completely understandable and admirable) sometime within next decade. Users would prefer it yesterday, but short of suing the company have no sway. Even other teams within the company have to resort to hacks to get feature working!

• No one cared to comment on my question about parent / child relations within read-only ref structs (theoretically, perfectly safe construct)

• No one provided any meaningful info on how this issue could be properly escalated to higher levels of decision makers to expedite its implementation and release.

Are there any work being done or scheduled regarding this topic (struct ref ...) in runtime or Roslyn? Is this tempest in a tea pot even necessary?

[jaredpar]

Are there any work being done or scheduled regarding this topic (struct ref ...) in runtime or Roslyn?

This was discussed for C# 9 but just fell off the priority list. It will be considered as part of C# 10. I'm the champion for the feature, have the rough design for it and related features, and once we get past the .NET 5 ship cycle will likely get it polished up for review.

No one provided any meaningful info on how this issue could be properly escalated to higher levels of decision makers to expedite its implementation and release.

Possibly because such channels don't really exist. GitHub is our primary vehicle for discussions and prioritization around features.

The rationale for why this can't be made public and what the path forward for proper inclusion in the ecosystem has been stated many times on this thread. I'm sorry that the answers aren't satisfactory but they are the answers to the best of our ability.

[GrabYourPitchforks]

@jaredpar I look forward very much to your proposal! :) Let us know what you need on the runtime + libraries side to pull it off.

[ENikS]

This was discussed for C# 9 but just fell off the priority list. It will be considered as part of C# 10. I'm the champion for the feature, have the rough design for it and related features, and once we get past the .NET 5 ship cycle will likely get it polished up for review.

Is there a good way to make sure it will not 'fell off the priority list' next time?

[jaredpar]

Is there a good way to make sure it will not 'fell off the priority list' next time?

Keep an eye on https://github.com/dotnet/csharplang/issues/1147. That is the main issue I'm using to track the request at the moment (there are actually many overlapping issues for this feature). Once the proposal is finished that issue will have an update to point at it and from there you can see the LDM discussions on it.

Probably won't make it up there for at least a few more weeks though.

Q: Well what if @jaredpar never actually writes a proposal for this?
A: Likely several people on the team will hunt me down, even during covid, and make me write it.

[ENikS]

@Sergio0694 I guess it is safe to close this issue now

@jaredpar

Well what if @jaredpar never actually writes a proposal for this?

Now, that we know where you live on GitHub, we will be haunting you as the shadow of Hamlet's father... :).

[tannergooding]

Likely several people on the team will hunt me down, even during covid, and make me write it.

Several of the features I want in are dependent on the feature, so I poke @jaredpar about it every know and again, as he can attest 😄

[terrajobst]

Surveillance footage from when we had a team room:

[jaredpar]

FYI: the initial proposal for allowing ref fields in C# is up for review on csharplang

https://github.com/dotnet/csharplang/pull/3936/

[GrabYourPitchforks]

Given that Jared is driving this from the C# side, is there anything left for this runtime issue to track? Will close this issue after a few days if there's no remaining work.

[tannergooding]

How discoverable are closed vs open issues?

I know, for example, they don't show by default in issue search so that might be a point towards keeping it open until the C# feature is implemented.

[tannergooding]

(We do have the tracking-external-issue label for such occasions)