Corefxlab: Close on the design of native-size numbers

[KrzysztofCwalina]

We worked on a design document for native-sized number types that we would like to add to the platform.

A draft of the design document has been just posted to https://github.com/dotnet/corefxlab/blob/master/docs/specs/nativesized.md

Feedback would be appreciated, especially on the “Design Considerations” section which describes open issues/questions/trade-offs.

[nietras]

The biggest issue with this is you cannot undo C# mapping of IL e. g. native int to IntPtr, how will this be adressed?

For further consideration see https://github.com/DotNetCross/NativeInts for how types could be implemented in IL.

[mellinoe]

We have considered adopting IntPtr and UIntPtr for these scenarios, but we decided against it because of compatibility reasons (it would require changing behavior of UIntPtr and IntPtr). We decided that separate types for integer and pointer scenarios is the best approach at this point.

I feel that this is skimmed over a bit too quickly; can we expand on what the specific incompatibilities are? Given that most/all existing interop mappings will just choose IntPtr or UIntPtr for these scenarios today, these new types at the very least will introduce a kind of "API schism" going forward.

[CyrusNajmabadi]

I see the value here. Though i question if it's necessary to have a specific language keyword for this. I think it would be fine if someone had to write IntN. And, if they really wanted to write 'nint', they could just say: "using nint = System.IntN".

[mellinoe]

I'm not sure about the general usefulness of FloatN. The usage of a typedef in a native signature (CGFloat is just a typedef for a 32- or 64-bit float) is a general problem not restricted to this Apple framework, and this proposal does not solve that; just one particular instance of it. Adding runtime support, language keywords, etc. for such an esoteric and unfriendly pattern feels like overkill to me.

Other discussions on this topic suggested putting this into a separate ("Apple-specific") library, since it is only useful for their unusual usage of a float typedef in public function signatures. Is that not an acceptable solution?

[VSadov]

to emit math with IntN , compiler would need to unwrap to the underlying values, do the math, and then wrap the result back into IntN. Wrapping back is easy since there is a conversion operator, but how wnwrapping would happen?

The proposal is a bit too vague on that.
Even if it is intrinsically convertible, compiler will need to emit something.

Do we have an IntPtr value field, ToIntPtr property, or something else?

Basically, what is the expected IL for the following:

C# nint x = 21; x = x + x;

[VSadov]

I am interested in more details about compatibility concerns.

Since the goal here is to allow efficient math in terms of native integers, just adding a bunch of compiler built-in checked/unchecked operators like nint + nint -> nint , nint + int -> nint, ... would seem to do the job most efficiently (in terms of emitted IL).
Compiler then can emit that directly as checked/unchecked math on native ints/uints
This approach works nicely for pointer types today.

Mapping to IntPtr would naturally "just work" with interop and existing APIs.
For example dynamic COM binder knows about native ints. It would not understand IntN, without servicing.

If the only concern is overload resolution between, say int and nint, there could be other solutions to keep the existing behavior. For example we can keep the types not convertible implicitly except when dealing with literals.

Basically, I see many advantages in mapping nint directly to IntPtr when efficiency is concerned, so I wonder what disadvantages push us to use the wrapper approach?

I do understand that for nfloat we need a wrapper, but for nint we might not need to.

[VSadov]

An alternative proposal for nint could be:

• map nint directly to IntPtr
• introduce built-in operators for IntPtr, including those that take int such as nint + int -> nint
  (emitted as plain add or add.ovf if in checked context)
• allow nint when indexing arrays and in pointer element accesses.
• introduce built-in explicit conversions from numeric integral types
  (emitted as conv.i , conv.u, conv.ovf.i or conv.ovf.i.un depending on context and operand's type)
• introduce built-in explicit conversions to numeric integral types
  (emitted the same as conversions from long, i.e. conv.ovf.u8 when converting to ulong in checked)
• do not introduce implicit conversions

(similarly nuint can be mapped to UIntPtr, but needs to use unsigned semantics for math and conversions)

The end result will be:

```C#

// Metadata: IntPtr M1(IntPtr[] arr, IntPtr pos, int offset)
nint M1(nint[] arr, nint pos, int offset)
{
// can add nint and int
var i = pos + offset;

// can use  nint as an array index
return arr[i];

}

// must cast 123 to nint explicitly for compat reasons.
M1(someArray, (nint)123, 45)

```

The advantage here is that such nint works right away with everything that understands IntPtr.
Including scenarios where values are boxed, used dynamically, passed to COM interop as Variants and so on...

[VSadov]

And to answer my question above - there are indeed problems with mapping nint directly to IntPtr in the way I've described.

Turns out IntPtr already defines some math operators and explicit conversions to/from ints and pointers. It probably felt like a good idea long time ago to add some math capabilities to this type, since C# would not provide any.
Now, when we would like to provide correct and efficient support via intrinsics, we will run into the risk of breaking some existing code.

Sigh...

[VSadov]

@KrzysztofCwalina - the example of UIntN in the proposal is missing implicit conversion that goes the other way. UIntN->UIntPtr

I assume that is how the underlying value is fetched when operating on the struct.

[jveselka]

public struct IntN : IComparable<IntN>, IEquatable<IntN>, IFormattable {
    public static IntN MaxValue { get; } // note that Int32 uses const 
    public static IntN MinValue { get; } 
    …
}

But IntPtr.Zero uses static readonly field. Is there a reason for using get-only properties instead?

[Drawaes]

Doesn't a static property like that if backed with a static readonly (which it should in this case) get basically collapsed to similar to a const by the hit anyway. If it's a property it allows for future behaviour change with no breaking of the api.

[benaadams]

Zero is constant; min and max value depend on native size which isn't known until runtime. (Though min value is a const for UIntN)

[KrzysztofCwalina]

@nietras, could you clarify the mapping issue? We map IntPtr to native int. We can map IntN to native int too. We don't map native int to BCL types except when here is metadata telling us what mapping is desired. Am I missing something? cc: @jkotas

@mellinoe, there is a sentence alluding to the incompatibilites, but I will expand it. Thanks.

@CyrusNajmabadi, without the alias, the types will forever feel like 2nd class citizens. But you are right that it's not absolutelly necessary, hence pri 2.

@mellinoe, we considered doing what you suggested with GCFloat, for the reasons you listed. But it seems like the cost of doing proper language support is not that high (once we do this for ints), the type is very important to Xamarin, and it future proofs us in the case the type becomes more widely used. cc: @migueldeicaza

@VSadov, the conversions to IntPtr/UIntPtr are real overloaded operators. See the asterix in the table. These will be used to unwrap.

@VSadov, there are conversions UIntN->UINtPtr in the table. Admitably in an awkward place: see the second table. BTW, this is where the asterix operators are, which are APIs (overloaded operators) for the unwrapping scenarios.

@zippec, as @benaadams said, some of the min/max values are not known till runtime. The others that can (e.g. UIntN.Zero) are properties for consistency with these that cannot. But we should consider making these consts. @jkotas?

[tannergooding]

@mellinoe, I would very much rather see the support added to System.IntPtr and System.UIntPtr as well, for all the various reasons.

However, @jaredpar raised a very good point offline that the forced 'checked' behavior for some of the existing code (constructors, and some of the conversions, and operators) makes it very difficult since that means we would either have to take a breaking change to remove the forced 'checked' behavior or we would need to have inconsistent behavior when dealing with IntPtr vs how we deal with everything else.

[tannergooding]

That being said, I think it would still be useful to finish fleshing out IntPtr and UIntPtr with all the appropriate operators (and doing so in a consistent manner with how things already are for the available surface area).

Regardless of whether the type was designed specifically for pointers (and I'm not sure I agree with this, since the remarks section and CLI spec both indicate otherwise), pointer arithmetic is a very valid and useful scenario which should be supported on the IntPtr type.

[Drawaes]

Agreed IntPtr.Add (mypointer, offset) is ugly and hoop jumping.

[nietras]

much rather see the support added to System.IntPtr and System.UIntPtr

I agree completely with this, with some considerations below.

We don't map native int to BCL types except when here is metadata telling us what mapping is desired. Am I missing something?

@KrzysztofCwalina Or perhaps I am missing something but given the following C# code:

    public struct nint
    {
        public IntPtr Value;

        public nint(IntPtr value)
        {
            Value = value;
        }
    ....

This is compiled into:

.class public sequential ansi sealed beforefieldinit DotNetCross.NativeInts.nint
       extends [System.Runtime]System.ValueType
{
  .field public native int Value

  .method public hidebysig specialname rtspecialname 
          instance void  .ctor(native int 'value') cil managed
  {
    // Code size       8 (0x8)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  ldarg.1
    IL_0002:  stfld      native int DotNetCross.NativeInts.nint::Value
    IL_0007:  ret
  } // end of method nint::.ctor

Notice how IntPtr is not used or referenced. Only built-in native int is used. Thus, IntPtr is simply speaking a placeholder for native int. The same goes for UIntPtr.

This works the other way too, of course, which can be seen in e.g. https://github.com/DotNetCross/NativeInts and of course the Unsafe class https://github.com/dotnet/corefx/blob/master/src/System.Runtime.CompilerServices.Unsafe/src/System.Runtime.CompilerServices.Unsafe.il but also in any other code using these types. IntPtr and UIntPtr as such do not exist in the generated code, as far as I can tell. Which is great from an efficiency point, but less than ideal from a naming consideration.

I believe we should simple accept that IntPtr and UIntPtr are defacto keywords for native int and unsigned native int, perhaps I am missing something in this, as I am not completely aware of why the IntPtr and UIntPtr types exist and there have not been made actual language keywords for these types as is the case for e.g. int and System.Int32.

Anyway, the naming with Ptr is less than ideal of course, but how can this be changed? I can't see how without breaking stuff... which means if we want, and I believe we do, an efficient (e.g. IL code size) and zero coupled way (e.g. not introducing new types that need to be referenced which means new "runtime"/framework versions etc. and thus no support for older runtimes) of using the IL native sized types, we have to use IntPtr and UIntPtr and then make these full fidelity types will all operators and so forth.

So I do not understand the "metadata" part of the question, perhaps this is where I am missing something? 😕

[VSadov]

@tannergooding - IntPtr/native int and UIntPtr/native uint are indeed just pointer-sized integer types. That is why there are signed/unsigned versions - to do signed/unsigned math.

However the fact of adding operators to those types a while ago has essentially closed the door to implementing the math support as compiler intrinsics now - due to compatibility concerns.

[mellinoe]

@mellinoe, we considered doing what you suggested with GCFloat, for the reasons you listed. But it seems like the cost of doing proper language support is not that high (once we do this for ints), the type is very important to Xamarin, and it future proofs us in the case the type becomes more widely used. cc: @migueldeicaza

I understand that reasoning, but I'm skeptical that we'll ever see other libraries have such a pattern. It seems extremely unusual to me, and I don't think I've encountered another library doing something like it. I'm also not aware of other languages or standard libraries that have this kind of "native float" type, whereas native integers and integer pointers are universal and standard. I'd probably feel comfortable with more examples of this pattern being used. Do we know of any others than CGFloat?

Related to FloatN: How is it constructed? Are there proposed literals being added to the language? I don't see conversions for it listed here; I guess there are implicit conversions from float and double?

[MichalStrehovsky]

How strongly do people feel about the naming of these? While I can see why things like Int32/UInt32 don't conform to the framework naming rules on multiple fronts (the reason I assume is that int is a concept well known to every programmer and it's nice to keep them short):

• DO NOT use abbreviations or contractions as part of identifier names
• DO choose easily readable identifier names
• DO favor readability over brevity.

I'm not aware of the N suffix being a standard in .NET for anything. Especially UIntN just feels like an unreadable soup of lower case and capital letters.

Why not just call them what they are? NativeInt/NativeUInt/NativeFloat? That way people don't have to spend time thinking why the N is a suffix in the proposed type name, but a prefix in the proposed language keyword.

[nietras]

@jaredpar raised a very good point offline that the forced 'checked' behavior for some of the existing code (constructors, and some of the conversions, and operators) makes it very difficult since that means we would either have to take a breaking change to remove the forced 'checked' behavior

@tannergooding Couldn't we just add support for specifying unchecked then? Like for int. The generated code is pure IL and doesn't reference the IntPtr reference impl as such, so supporting unchecked/checked should be possible, although perhaps not with the same defaults as int.

[jaredpar]

Couldn't we just add support for specifying unchecked then? Like for int.

The design explicitly calls for both checked and unchecked support to match the existing integral types. Need to support this case and IntPtr can't do that without breaking compat in some way due to the existing operators.

[tannergooding]

@jaredpar, its a bit unprecedented (at least to the C# compiler, but not to compilers in general), but would it be crazy to have a compiler switch that says to ignore the existing operators defined by the framework and to emit pure add/add.ovf instructions instead?

I believe that would let existing code continue to work and let new consumers, who explicitly want checked/unchecked behavior to opt-in.

My thinking is that, overall, it may be worthwhile since we have 17 years worth of existing APIs/interop code (including in the framework itself) where we exclusively use IntPtr/UIntPtr.

[jaredpar]

switch that says to ignore the existing operators defined by the framework and to emit pure add/add.ovf instructions instead?

Imagine that command line option exists. How can I sanely review code which uses IntPtr now? It's impossible to understand the semantics unless I understand every single compilation in which it occurs. That is more of a problem now with the prevalence of shared projects where code is commonly used in multiple compilations.

Astute readers may note: wait didn't @jaredpar just argue specifically against having a /checked command line option? Yes I did. That option makes it impossible to correctly interpret virtually any arithemetic operation in your source code. It's part of the reason it has so little usage.

My thinking is that, overall, it may be worthwhile since we have 17 years worth of existing

Disagree strongly. A command line switch which changes the semantic behavior of the compiler is a bad feature. It makes it impossible to read code and understand what it does.

[tannergooding]

I understand your viewpoint, but am not completely in agreement here. A large number of compilers have just such behavior for math (especially with regards to floating-point arithmetic -- /fp:fast for example). #pragma directives are another example where compilers frequently modify behavior of code to suit particular needs.

[Drawaes]

And it's a nightmare to manage. Your writing a precision sensitive finance library and a mistake is made in compilation, tracking that back is nigh on impossible. The nice thing about .net today is if I write a Monte Carlo and run it on an old çomputer in linux or on a Mac or whatever the numbers will match (with correct seeding). I also think this is why that native float is bad to have outside a specialized graphics lib. The numerical ramifications of flipping from a float to a double can be huge.

[jaredpar]

A large number of compilers have just such behavior for math (especially with regards to floating-point arithmetic -- /fp:fast for example

Sure. And my feedback still applies. The presence of such switches make it harder to interpret what code is doing. It's entirely relying on the decision of an ambient authority. The decision which is often difficult to determine (saying this from experience trying to figure out how a series of makefiles translate into a final command line for the compiler).

It also makes the code significantly less portable. Because the code and the ambient floating point option must be carried together.

pragma directives are another example where compilers frequently modify behavior of code to suit particular needs.

I dislike #pragma for similar reasons but it at least is local to the file. Hence it's easy to determine for a casual code reviewer in a web browser what the behavior of the code in question is. After a lot of scrolling around at least 😦

[benaadams]

Native float is used for graphics https://developer.xamarin.com/guides/cross-platform/macios/nativetypes/#CoreGraphics_Types

[Drawaes]

Yup and I am saying that is where they should stay. My fear is they will leak out into other parts of the framework if they are a general type. Also have they ever been used outside Apple specific API's ? Every GPU I have worked with you had to explicitly call with singles or doubles it didn't just change depending on the platform.

[Drawaes]

Not to mention is this even an issue for IOS soon?
Apple Drop support for 32bit

[benaadams]

Not to mention is this even an issue for IOS soon?

Don't know, might make it worse as the Windowing measures on iOS will be 64 bit floats vs 32 bit floats on Windows which makes it more problematic to make a unified Windowing api between them; but xamarin folks would know more.

[nietras]

IntPtr can't do that without breaking compat in some way due to the existing operators.

@jaredpar I understand that any method or operator on e.g. IntPtr forward to the IntPtr implementation e.g. for the same type as above:

        public unsafe static nint operator +(nint l, int r)
        {
            return l.Value + r;
        }

yields the following IL:

  .method public hidebysig specialname static 
          valuetype DotNetCross.NativeInts.nint 
          op_Addition(valuetype DotNetCross.NativeInts.nint l,
                      int32 r) cil managed
  {
    // Code size       18 (0x12)
    .maxstack  8
    IL_0000:  ldarg.0
    IL_0001:  ldfld      native int DotNetCross.NativeInts.nint::Value
    IL_0006:  ldarg.1
    IL_0007:  call       native int [System.Runtime]System.IntPtr::op_Addition(native int,
                                                                               int32)
    IL_000c:  call       valuetype DotNetCross.NativeInts.nint DotNetCross.NativeInts.nint::op_Implicit(native int)
    IL_0011:  ret
  } // end of method nint::op_Addition

And the IntPtr::op_Addition is then defined as the following on 64-bit:

.method public hidebysig static specialname native int 
    op_Addition(
      native int pointer, 
      int32 offset
    ) cil managed 
  {
    .custom instance void System.Runtime.ConstrainedExecution.ReliabilityContractAttribute::.ctor(valuetype System.Runtime.ConstrainedExecution.Consistency, valuetype System.Runtime.ConstrainedExecution.Cer) 
      = (01 00 02 00 00 00 01 00 00 00 00 00 ) // ............
      // int32(2) // 0x00000002
      // int32(1) // 0x00000001
    .custom instance void System.Runtime.Versioning.NonVersionableAttribute::.ctor() 
      = (01 00 00 00 )
    .maxstack 8

    IL_0000: ldarga.s     pointer
    IL_0002: call         instance int64 System.IntPtr::ToInt64()
    IL_0007: ldarg.1      // offset
    IL_0008: conv.i8      
    IL_0009: add          
    IL_000a: newobj       instance void System.IntPtr::.ctor(int64)
    IL_000f: ret          

  } // end of method IntPtr::op_Addition

But couldn't this instead map directly to IL instructions on native int instead? I understand this circumvents any changes to IntPtr in the future, but any such change would be very bad anyway I assume, so what are the drawbacks of simply sidestepping this and then support checked/unchecked by emitting IL directly instead? Most of the functionality on IntPtr can be implemented directly as IL simply side-stepping the methods, but keeping the existing behaviour, but making it possible to modify this with local checked/unchecked scopes.

[jaredpar]

@nietras

The problem is that IntPtr has a set of existing operators today which have defined checked / unchecked semantics. These cannot change due to back compat concerns. These operators include the set this design explicitly want to have checked / unchecked semantics.

Concrete example:

c# var ip = new IntPtr(long.MaxValue); var val = unchecked { (int)ip };

The design explicitly calls for the second statement to have unchecked semantics: in both the addition operator and the conversion from a native int to int. This cannot be though because the conversion from IntPtr to int already has defined checked semantics due to the presence of the operator

• on 32 bit the conversion is unchecked
• on 64 bit the conversion is checked

The 32 bit case doesn't matter here because it would actually throw on the first statement. The second statement though is interesting. The (int)ip code could be executed today and would do so in a checked fashion which causes an exception. But following the spec this would now silently truncate the value. That's a compat break.

This is just one simple example. But it demonstrates why we can't both give IntPtr full arithmetic support, including checked / unchecked support and maintain compat.

I agree it's frustrating. It would be great if we'd never defined any operators here and the type could simply be extended to support these scenarios. However that's not the case and compat is king.

[nietras]

@jaredpar I just realized this too after my comment. I think I understand
now, and agree that making such a breaking change is bad.

Had been hoping we could find a good compromise that would not involve
adding new types, but seems less and less likely. If only native int had
not mapped directly to IntPtr in signatures (e. g. method parameters),
but the past is the past.

On Apr 18, 2017 23:03, "Jared Parsons" [e-mail] wrote:

@nietras https://github.com/nietras

The problem is that IntPtr has a set of existing operators today which
have defined checked / unchecked semantics. These cannot change due to back
compat concerns. These operators include the set this design explicitly
want to have checked / unchecked semantics.

Concrete example:

var ip = new IntPtr(long.MaxValue);var val = unchecked { (int)ip };

The design explicitly calls for the second statement to have unchecked
semantics: in both the addition operator and the conversion from a native
int to int. This cannot be though because the conversion from IntPtr to
int already has defined checked semantics due to the presence of the
operator

• on 32 bit the conversion is unchecked
• on 64 bit the conversion is checked

The 32 bit case doesn't matter here because it would actually throw on the
first statement. The second statement though is interesting. The (int)ip
code could be executed today and would do so in a checked fashion which
causes an exception. But following the spec this would now silently
truncate the value. That's a compat break.

This is just one simple example. But it demonstrates why we can't both
give IntPtr full arithmetic support, including checked / unchecked
support and maintain compat.

I agree it's frustrating. It would be great if we'd never defined any
operators here and the type could simply be extended to support these
scenarios. However that's not the case and compat is king.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefxlab/issues/1471#issuecomment-294981967,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKTG7-wXGjSXnsu6fXKki7nXqyT1E5Mnks5rxSUbgaJpZM4M_UDd
.

[benaadams]

Also have they ever been used outside Apple specific API's ? Every GPU I have worked with you had to explicitly call with singles or doubles it didn't just change depending on the platform.

Nevertheless it is a platform where .NET runs and it changes or differs from other platforms; like x86 vs x64 for pointers

Size of floating-point data types in OS X and iOS

type | ILP32 size | LP64 size
--------|------------|----------
float | 4 bytes | 4 bytes
double | 8 bytes | 8 bytes
CGFloat | 4 bytes | 8 bytes

[Drawaes]

Agreed, but it's a platform that is about to remove support for the 32bit and harmonize on 64bit, so the float native's use case is for a single platform that isn't about to be supported anymore.

[benaadams]

so the float native's use case is for a single platform that isn't about to be supported anymore.

No; because the depreciation of 32bit means macOS and iOS will always use 64 bit floats and Windows will always use 32 bit floats (outside of explicit fixed sized double and float); so if you want to create a platform neutral api between both you still need a data type which changes size based on platform.

[OtherCrashOverride]

My interpretation of this proposal is to basically add a C "size_t" type to .Net.

Is there a reason that a native type to represent C "long" is not included in the proposal? This type differs on LP64 and LLP64 (everyone else vs windows).

Also, as mentioned in another thread. There is no such thing as a "native float". Both single and double precision floats are native (implemented in hardware) on all supported platforms including iOS.

[MichalStrehovsky]

so if you want to create a platform neutral api between both you still need a data type which changes size based on platform

@benaadams Windows will use 32bit floats even on x64, so you can't use FloatN for such API anyway.

[mellinoe]

@benaadams I don't think that makes sense, and I don't think that is the motivation for this scenario. If you're creating some sort of abstraction layer for two different platforms, you should create a unified representation within your API and convert it to platform-specific representations when necessary. The main impetus for FloatN seems to be the desire to PInvoke to 32- or 64-bit CoreGraphics functions using the same structure.

@OtherCrashOverride Agreed, I think the term "native float" is a bit confusing. What is described here is a "pointer-sized float". Outside of this discussion, I've not heard this term used.

[mellinoe]

Windows will use 32bit floats even on x64, so you can't use FloatN for such API anyway.

That seems very surprising to me, and doesn't really seem to be reflected in the proposal document (at least not clearly). What exactly is FloatN, then?

EDIT: Never mind, I think I misinterpreted what you said. FloatN will be 64-bit, but Win32 API's still generally take 32-bit floats.

[OtherCrashOverride]

FWIW, an example solution to the CGFloat issue would be to use System.Double as the type exposed in .Net APIs and wrap the PInvoke.

using CGFloat = System.Single;

public void ExampleCall(double value)
{
   CGFloat apiValue = value;
   ExampleCall_PInvoke(apiValue);
}

With 32bit iOS already deprecated, is there some other use case that warrants the proposed FloatN? This seems like a lot of effort to support a "dead" platform.

[benaadams]

Windows will use 32bit floats even on x64, so you can't use FloatN for such API anyway.

Not sure how that is relevant? 3 things (at least) are baked into the runtime; bittage (e.g. 32 vs 64), OS (e.g. Win/Mac/Linux) and endianness (big/little). The size of the float used in UI graphics is determined explicitly by the runtime, as it is for pointers and x86 vs x64.

Agreed, I think the term "native float" is a bit confusing. What is described here is a "pointer-sized float".

Its a float size determined by OS+bittage.

you should create a unified representation within your API and convert it to platform-specific representations when necessary

Its the basic building block of all the UI graphics. So what do you store it as? Lowest common denominator in float then always upcast to double on macOS/iOS on every call? As a struct that wraps an IntPtr then cast from int to double or float in a behind the scenes native platform shim for all platforms?

What's the C# struct look like?

With 32bit iOS already deprecated, is there some other use case that warrants the proposed FloatN? This seems like a lot of effort to support a "dead" platform.

5 year old previous macOS has a 32 bit only mode and 64 bit macOS still runs 32 bit apps; when it then uses the 32 bit Cocoa api.

While I hear your questions "Why would you ever use a 5 year old OS?" and "but why would you develop a 32 bit app for a 64 bit OS?". For those answers you might want to ask the current users of Windows 7 and the developers of Visual Studio. 😝

[benaadams]

Actually looks like Windows is 64 bit e.g. System.Point is (Double, Double); however 32 bit android looks like its 32 bit floats.

[OtherCrashOverride]

Its a float size determined by OS+bittage.

I think this is the fallacy. It is NOT determined by OS+bittage. It is an API choice for a single component: CoreGraphics. As proof of this assertion, the OpenGL API, that also uses floating point, does not change based on OS+bittage for Apple platforms.

Its the basic building block of all the UI graphics. So what do you store it as?

As stated previously, System.Double makes the most sense. Its simply a matter of a wrapper using 'double' with two p/invoke backends: CoreGraphics32 and CoreGraphics64.

What's the C# struct look like?

struct CGFLOAT
{
#if APPLE_32BIT
   float value;
#else
   double value;
#endif

  public CGFloat ApiOperationsAreExpressedInTermsOfCGFloatItself()
  {
  }

  public static CGFloat operator +(CGFloat c1, CGFloat c2)
  {
  }

  // Explicit conversions to System.Single/Double here, etc.
}

While I hear your questions "Why would you ever use a 5 year old OS?"

The question is actually to quantify the merit of architecting new features for obsolete platforms. What is the impact of this feature going forward? Adding support to the runtime and languages is of non-trivial impact.

[sharwell]

@jaredpar It appears people are assuming that operators and/or conversion operators need to be defined in IL in order for this feature to be implemented. I do not believe that is the case. I would expect the C# compiler to emit code which does not use these operators even if they are defined, which certainly addresses the case of unchecked((int)nativeInt), which would emit something like:

ldarg.0
conv.i4

My interpretation of ECMA-335 §I.8.2.2 is the virtual machine already knows to load values of type IntPtr and UIntPtr onto the evaluation stack as "native int" and "native unsigned int" respectively, allowing the defined IL operators for these values to be used without method calls.

tl;dr For the case of native int and native unsigned int, the feature can be implemented in its entirety within the C# compiler; the standard library already contains the necessary supporting code and has since at least .NET 2.0.

[tannergooding]

@sharwell, that is what I (and others) would like to happen. However, doing that may break back-compat, since some operators are already defined and may already be in use.

Recompilation of existing source code with the new compiler could cause it to behave differently, since the new code would follow checked/unchecked rules where the old code wouldn't (although, as I understand it, overload resolution changes and the like can also be a concern under recompilation with a new compiler...)

That being said, there is at least one other type where we ignore predefined operators, I'll see if I can find the section...

[sharwell]

@tannergooding The following are cases where existing operators are affected.

1. When running on 32-bit platforms, the following operators are affected. The operators in IL are always checked, while the language feature for the same conversion would be unchecked by default. This case is likely to affect existing code.

   explicit operator IntPtr(long value)
   explicit operator UIntPtr(ulong value)
   

2. When running on 64-bit platforms, the following operators are affected. The operators in IL are always checked, while the language feature for the same conversion would be unchecked by default. This is the most likely case which would break existing code.

   explicit operator int(IntPtr value)
   explicit operator uint(UIntPtr value)
   

3. The following operator is affected within an overflow-checking context. The IL operator is never checked, while the language feature for the same operators would be unchecked by default but could also appear in a context where overflow checking occurs. In most cases where a behavior change is observed, the new behavior would actually be what the user thought was occurring the whole time.

   IntPtr operator +(IntPtr pointer, int offset)
   IntPtr operator -(IntPtr pointer, int offset)
   UIntPtr operator +(UIntPtr pointer, int offset)
   UIntPtr operator -(UIntPtr pointer, int offset)
   

:memo: If we fail to define this feature using the existing IntPtr and UIntPtr types, a substantial amount of totally avoidable runtime overhead will be incurred. The specification for the runtime explicitly states that the type IntPtr is the metadata representation of "native int" and UIntPtr is the metadata representation of "native unsigned int". No conversions or calls are necessary in order to use these types in this manner, but in any other case we will not be able to leverage IL instructions which operate directly on these types. Considering the applications where compiler support for native-sized integers is most likely to be used, I do not believe any solution which uses alternative types will meet the needs of these users.

[sharwell]

💭 It would be relatively straightforward to introduce a compiler warning for an unchecked explicit conversion (cast) from native-sized integer to a 32-bit integer which did not appear in a unchecked(...) expression. This would somewhat mitigate the second problem, and a similar warning would cover the first problem.

The third problem is arguably not going to be a problem. The language expression affected is something like this:

IntPtr pointer = ...;
int offset = ...;
IntPtr newPointer = checked(pointer + offset); // currently checked has no effect

A somewhat more subtle case would be a checked block, or when compiling with /checked.

[tannergooding]

I've tried arguing that same point a number of times :)

Feel free to try and convince @jaredpar (and the ret of the C# LDM) though

[sharwell]

@tannergooding I was late to the discussion and missed your previous comments. Sounds like we're on the same page.

[jaredpar]

@tannergooding, @sharwell

Bringing back the concrete sample which demonstrates compat concerns around re-using IntPtr

c# var ip = new IntPtr(long.MaxValue); var val = unchecked { (int)ip };

This is code that compiles today with a defined behavior that would absolutely change if we had IntPtr respect checked and unchecked. Back compat changes of this nature cannot be taken unless there is an enormously strong reason to do so. The awkwardness of adding conversions between old and new code doesn't come close to meeting that bar.

It would be relatively straightforward to introduce a compiler warning for an unchecked explicit conversion (cast) from native-sized integer to a 32-bit integer which did not appear in a unchecked(...) expression.

This is also a back compat break which would cause a significant amount of existing code to stop compiling. I understand the recommendation is for a warning, not an error, but from the perspective of compiler compat there is no difference.

A good portion of customers use /warnaserror and others simply don't distinguish between the two. If the compiler issues a message, the build is broken and if it issues new warnings then the new version of Visual Studio has broken their code. This is a lesson we've learned pretty much every time we forget it.

From the perspective of the compiler and language compat is king. It's only broken under very specific circumstances, and generally when there are no other options available to us. For a feature as old as this there is not a chance it will break customers, there is a guarantee. Justifying that break, especially in the face of a design that does not have a compat issue, is a non-starter for the compiler.

[sharwell]

This is also a back compat break which would cause a significant amount of existing code to stop compiling. I understand the recommendation is for a warning, not an error, but from the perspective of compiler compat there is no difference.

For a feature as old as this there is not a chance it will break customers, there is a guarantee.

I totally agree regarding both the importance of compatibility and the likelihood of new predefined operators proving problematic. The leadership's view of the former is one of the reasons why why I hold C# in such high regard.

Justifying that break, especially in the face of a design that does not have a compat issue, is a non-starter for the compiler.

I don't consider the metadata operators to be a design which meets this description, primarily because I do not believe it adequately addresses the needs of users who want compiler support for native-sized integers.

However, I would consider an approach reminiscent of /unsafe to be a non-breaking change (i.e. a new command line argument which enables predefined operators for IntPtr and UIntPtr at the language level in the compiler, along with the resulting behavior changes).

[benaadams]

However, I would consider an approach reminiscent of /unsafe to be a non-breaking change (i.e. a new command line argument which enables predefined operators for IntPtr and UIntPtr at the language level in the compiler, along with the resulting behavior changes).

Trouble is that's project level not local level like unsafe,fixed, checked and unchecked are; might also be using IntPtr and UIntPtr for pointers not just native ints.

[CyrusNajmabadi]

/unsafe feels like a very different beast to me. It does not change the semantics of existing constructs. It just states if something is allowed or not. i.e. int*, or stackalloc, or fixed don't change meaning if i have /unsafe of not, it just changes if the compiler allows me to use those or not.

That doesn't sound like what would happen if i had a /newintptrsemantics flag. Now it would be the case that i could have code that was completely identical, but would behave differently. I'd see IntPtr and have no idea what that meant until someone told me precisely how the binary would be compiled at the end of the day.

--

Aside#1: i already find it difficult to know how code will eventually be compiled (as compiler flags can be so far removed thanks to things like .targets). I don't know how I, as a developer, coudl go from looking at code to answering the question "how is this IntPtr going to actually operate at runtime?". That's something i would like to very much avoid.

Aside#2: I dislike that i don't know how arithmetic will work in C# for this same reason. Was /checked supplied or not? Who knows!

[sharwell]

@benaadams

Trouble is that's project level not local level like unsafe, fixed, checked and unchecked are; might also be using IntPtr and UIntPtr for pointers not just native ints.

For the projects that want to use native-sized arithmetic, I'm not sure this would actually be a problem. It's essentially a compiler switch which enables the use of targeted IL instructions in specific scenarios, where the operators already exist but aren't usable within C#.

@CyrusNajmabadi

/unsafe feels like a very different beast to me. It does not change the semantics of existing constructs. ...

True, but remember my claim is not based on this being an ideal outcome, but rather based on identifying an outcome which meets the needs of users who want to use native-sized integer arithmetic while also considering the needs of users who wrote code before these operators existed in C#. I would not claim that this idea is the only way to resolve the situation, but so far I have not seen a proposal which addressed the needs of both sides to this level, especially considering this approach also offers minimal (zero) impact on target framework/distribution.

[jaredpar]

@sharwell

This github thread is long so wanted to point back to my feelings on a global switch

https://github.com/dotnet/corefxlab/issues/1471#issuecomment-294959689

In summary, I'm very opposed to adding global switches / attributes that affect semantic behavior. It makes it impossible to understand how the code I'm writing will execute. Nor can I review code and understand it's meaning. Instead I have to consider all of the compilations in which it will be used to understand it's behavior. That's a bad way to code.

[tannergooding]

I would think most users writing native int arithmetic (which will be a minority of users who are writing interop code) will end up using the switch to take on the "desired" semantics and the only people who will use the old behavior is legacy code. So I don't really expect much confusion here...

And I do realize this is a battle I have already lost :)

[sharwell]

@jaredpar

In summary, I'm very opposed to adding global switches / attributes that affect semantic behavior. ... That's a bad way to code.

I agree here too. Three things I'm opposed to in order of most opposed to least opposed are:

1. Changing the language semantics in a way that breaks existing code
2. Implementing a language feature which does not address the needs of its primary target audience
3. Implementing a compiler flag which affects the language semantics

So aside from the obvious possible outcome of not supporting native-sized arithmetic in C#, the flag is the least of the evils presented thus far, in my opinion of course. :smile:

[nietras]

@jaredpar to keep brainstorming I have yet another idea. We add nint and nuint as new keywords only. To C#. These keywords map directly to native int and unsigned native int. They do not exist as separate types. And thus are source code only.

Any use of e.g. nint in API places e.g. members, method parameters/returns will thus map directly to native int and thus for a consumer of the API be IntPtr. We simply allow a developer to define in source code that he wants to use native int directly with emitted code being in IL only, no method calls or such, but code as would be generated when using int. The new keywords would allow for checked/unchecked support as for any other built-in primitive.

To sum up this idea proposes:

• Introduce nint/nuint as new keywords.
• These keywords map directly to the existing IL types native int and unsigned native int
• All operators available in IL for these are supported using the keywords, with defaults matching normal behaviour.
• The keywords respect normal checked/unchecked behaviour.

Pros

• Allows for writing optimally efficient code.
• Does not introduce any new types or dependencies.
• Does not introduce any new compiler flags.

Cons

• Introduces new keywords that might break existing code.
• Won't resolve the naming issue with IntPtr/UIntPtr in signatures.

If compatibility and how this would introduce breaking changes is a big concern, one could imagine "namespaced" keywords, a new concept.

Basically, this simply introduces IL equivalent keywords. Boxing an nint will be a IntPtr object. As such the type mapping doesn't change. It is only the generated code that changes.

Examples:

public nint Mid(nint a, nint b)
{
    return (a + b) / 2; // Maps directly to native int IL instructions
}

// Any consumer of this will see this as
public IntPtr Mid(IntPtr a, IntPtr b);

checked/unchecked is respected:

public nint Mid(nint a, nint b)
{
    return unchecked((a + b) / 2); // Maps directly to native int IL instructions without overflow detection etc.
}

A modification of this idea, would be to only allow nint/nuint in local function scopes so that these are not exposed in source code on API boundaries either to make this as explicit as possible.

[tannergooding]

I think the second.(boxing) would work, since you would need to work with existing IntPtr signatures.

I also think having a contextual keyword would work here. That is, you would have nint in addittion to something like checked/unchecked.

```
IntPtr UncheckedAdd(IntPtr left, Int right)
{
unchecked nint
{
return left + right;
}
}

IntPtr CheckedAdd(IntPtr left, Int right)
{
checked nint
{
return left + right;
}
}

[tannergooding]

Maybe also have a pragma directive for controlling arithmetic for the entire file #pragma unchecked nint and #pragma checked nint, and which would adjust the behavior (visibly) for that file, allowing users to type less code overall.

[sharwell]

The more I look at the affected scenarios, the more this is reminding me of the foreach change made in the past for captured locals. It's made me ponder an approach like the following:

1. Update the project system such that <LangVersion> defaults to 7 (or 7.x) for future releases. Update the project templates to set the language version to the latest known version at the time the project is created.
2. Create an IDE experience which allows users to easily update the language version for their project if they start using newer features, or if they open an old project in a new version of Visual Studio which supports C# 8+.
3. Establish a policy of providing a set of analyzers (disabled or hidden by default) which serve as migration aids to alert users to potential breaking changes.

The experience which updates a project to the latest language version could automatically query results from the migration aid analyzers to alert users to changes in advance.

This solution has additional requirements at the IDE level, but provides the following improvements:

• Allows for the optimal language improvement going forward (new feature is always enabled for C# 8+ scenarios, with consistent behavior across integer types)
• Restricts the possibility of silently breaking changes to cases where users are invoking csc.exe directly (outside the project system).
• Reveals cases where behavior would be different between C# 7- and C# 8+.

[jaredpar]

@tannergooding, @sharwell

I understand your point of view here and have considered it several times. But my position remains the same. The breaking change, that is fundamentally necessary to meet the design here and re-use IntPtr doesn't come close to meeting the bar. Especially in the face of a proposal that requires no breaking changes.

[KrzysztofCwalina]

Why not just call them what they are? NativeInt/NativeUInt/NativeFloat? That way people don't have to spend time thinking why the N is a suffix in the proposed type name, but a prefix in the proposed language keyword.

I think it would be good to follow the [U]Int naming pattern. I agree that "N" is cryptic. I could change the names to UIntNative. I would not like to change them to NativeUInt as it would not follow the existing naming pattern for ints.

Implementing a language feature which does not address the needs of its primary target audience

@sharwell, I would like to understand: which feature does not address the needs of its target audience, and why? Or the comment was just general one?

I understand your point of view here and have considered it several times. But my position remains the same. The breaking change, that is fundamentally necessary to meet the design here and re-use IntPtr doesn't come close to meeting the bar. Especially in the face of a proposal that requires no breaking changes.

Yeah, totally agree. I am not really sure why we would consider any breaking changes here. The cost of the new types (conceptual and work) is really not that high. nfloat discussion is more interesting (IMO); I can see the arguments about Apple moving away from 32-bits, but I would like to wait for @migueldeicaza to chimne in. He felt very strongly we need this type as a first class supported type.

[OtherCrashOverride]

Is there a reason that a native type to represent C "long" is not included in the proposal? This type differs on LP64 and LLP64 (everyone else vs windows).

The issue for this is here:
https://github.com/dotnet/coreclr/issues/2426

It would seem directly related to the work this discussion is tracking.

[MichalStrehovsky]

The cost of the new types (conceptual and work) is really not that high

Let's just hope we will not end up having to publish a table with guidelines on what type to use when because the fear of breaking people will make us introduce a new type whenever the existing doesn't fit. </snark>

[sharwell]

@KrzysztofCwalina

which feature does not address the needs of its target audience, and why? Or the comment was just general one?

ECMA-335 defines a mapping from "native int" to System.IntPtr and from "native unsigned int" to System.UIntPtr, and provides a variety of IL instructions dedicated to operations on these types. Any proposal which does not map native-sized integers to these types precludes the use of these instructions, incurs fairly substantial overhead in the resulting IL, and increases demands on the optimizing JIT in the event its even possible to achieve the performance of supporting the native-sized integer IL instructions directly.

[Drawaes]

I don't really have a concern around the ints but the floats I have big concerns around. As said before there is no native float type on most processor or os arcitectures. Intel supported 80bit floats for extended precision from the 8087 math co processor. So what is a native float on an Intel cpu on a 32bit os. Especially if as Ben says 16bit float is the "native" win64 measurement (or is it?). I guess we need the ios folks to comment.

[KrzysztofCwalina]

@sharwell, IntPtr will keep mapping to IL native int directly. IntNative will be a struct. It will be unpack (through cast to IntPtr which never fails) to IL native int explicitly before doing arithmetic operations. I talked to @jkotas about it and he asserted that JITs either already can optimize the overhead away or it will be easy to teach the JIT to do it.

cc: @russellhadley

[tannergooding]

I'm still not sure why something like checked IntPtr { } and unchecked IntPtr {} blocks would not solve the problem here (@jaredpar).

It would seem that it solves the concern over breaking backwards compat and that it solves the issue that a command line switch had of users not being able to easily recognize that the code has a different behavior.

The only "concern" I see with having a new "block" type is that users who are using native integer arithmetic (which is already expected to be a minority of users writing interop code or base level framework libraries) would have to be slightly more verbose. However, that could be solved by providing a #pragma statement to enable or disable the functionality at the file level (which also clearly calls out the behavior is being changed for that section of code).

Not only would something like this solve the issue, but it does it in a way that can be consumed on all versions of the framework (net20-net47 and ns1.0-ns2.0) with no required runtime changes, no new types, no new special optimizations required in the JIT, etc...

[jaredpar]

I'm still not sure why something like checked IntPtr { } and unchecked IntPtr {} blocks would not solve the problem here

Part of the problem is that this flag would control more than checked / unchecked. In order to a) use IntPtr and b) maintain compat then it needs to be understood there are actually 3 different modes for arithmetic / cast operations to address:

• checked: all operations on IntPtr are checked just as they are for int, long, etc ...
• unchecked: all operations on IntPtr on unchecked just as they are for int, long, etc ...
• compat: all operations on IntPtr that work today continue to work without change

Overall this means the compiler, and end developers, needs to consider four different modes for arithmetic operations: checked compat, checked noncompat, unchecked compat and unchecked noncompat.

Note that these modes can actually effect both whether or not the operation overflows and the actual mathematical output of the evaluation.

Here is an example which demonstrates the subtleties of these different interactions:

c# int x = 42; IntPtr ip = new IntPtr(long.MaxValue); IntPtr ip2 = ip + 42;

This is interesting because it both is a checked operation on 64 bit (irrerspective of the compiler checked state). Additionally the math is done in terms of integers. If this code was compiled under unchecked noncompat it would both not overflow and do math in terms of longs.

Hence the unchecked flag is sneakily controlling my mathematical output, not just my overflow.

[sharwell]

@KrzysztofCwalina The source alias approach works within the bounds of a single assembly, but causes problems when operating on results returned from a referenced assembly.

Example:

Project A:

public static class MyType {
  public static nuint MyMethod() => 0;
}

Project B:

nuint value = checked(MyType.MyMethod() - 1);

Since the compilation of project A turns the result of MyMethod from nuint to UIntPtr, the operation in project B ends up ignoring the checked expression and calling the operator defined on UIntPtr.

[tannergooding]

@jaredpar: yes, this code is ambiguous:

```C#
int x = 42;
IntPtr ip = new IntPtr(long.MaxValue);
IntPtr ip2 = ip + 42;

and is equivalent to both:
```C#
unchecked
{
    int x = 42;
    IntPtr ip = new IntPtr(long.MaxValue);
    IntPtr ip2 = ip + 42;
}
````
and
```C#
checked
{
    int x = 42;
    IntPtr ip = new IntPtr(long.MaxValue);
    IntPtr ip2 = ip + 42;
}

However, this clearly states the intention:
```C#
checked IntPtr
{
int x = 42;
IntPtr ip = new IntPtr(long.MaxValue);
IntPtr ip2 = ip + 42;
}

```C#
unchecked IntPtr
{
    int x = 42;
    IntPtr ip = new IntPtr(long.MaxValue);
    IntPtr ip2 = ip + 42;
}

Having an analyzer that calls out uses of IntPtr arithmetic in a regular checked/unchecked (or default) context would help remove the ambiguity in the first case and would be useable by users who know they are writing IntPtr arithmetic and are wanting the new behavior.

The only operations that would exist in the regular checked/unchecked/default context are the existing ones (constructors, cast, and the existing operators defined in the framework). The new ones would only be available in the checked/unchecked IntPtr contexts and can be highlighted as such by the compiler (it is already an error today, so give a better error indicating how to consume them).

[tannergooding]

@jaredpar, I meant ambiguous as to how you referred.

This is interesting because it both is a checked operation on 64 bit (irrerspective of the compiler checked state). Additionally the math is done in terms of integers. If this code was compiled under unchecked noncompat it would both not overflow and do math in terms of longs.

Hence the unchecked flag is sneakily controlling my mathematical output, not just my overflow.

That is, it has different behaviors on 32 vs 64-bit and users may not understand what it is doing (users may expect that the operations are checked/unchecked as they declared it). We should probably have an analyzer that points out this issue in the first place.

However, having the compiler ignore the pre-existing operators/constructor if the code is written in a unchecked/checked IntPtr block removes all the ambiguity and gives consistency. It clearly states the user is aware of what they want and they want the compiler to respect it. It removes the need for runtime changes, doesn't break back-compat, and allows users to easily follow/review the code.

[jaredpar]

@tannergooding

yes, this code is ambiguous:

This code has well defined semantics and executes consistently across all applicable runtimes. It is not ambiguous in any way.

PS: sorry for hitting close a minute ago.

[tannergooding]

This code has well defined semantics and executes consistently across all applicable runtimes.

Except for the different behavior on 32-bit vs 64-bit platforms. Where-as supporting a checked/unchecked IntPtr block makes it have consistency on all runtimes/platforms/bitness.

[jaredpar]

@tannergooding

That is, it has different behaviors on 32 vs 64-bit and users may not understand what it is doing

As I've noted, the examples I provided all have well defined behavior and consistent runtime semantics. The possibility that some users aren't aware of them isn't really relevant. There are certainly users out there who are well aware of them and write code that depend on the exact semantics they provide today.

It clearly states the user is aware of what they want and they want the compiler to respect it.

It makes it clear about the checked, unchceked nature. It in no way alerts me that you've also possibly changed the base type of my arithmetic.

Except for the different behavior on 32-bit vs 64-bit platforms

The behavior for this code on 32 / 64 bit is well defined and has been from the start.

Where-as supporting a checked/unchecked IntPtr block makes it have consistency on all runtimes/platforms/bitness.

The behavior for IntPtr with checked / unchecked is just as "odd" as IntPtr is. Both have quirks that have to be considered:

checked { 
  IntPtr ip1 = int.MaxValue;
  IntPtr ip2 = int.MaxValue;
  IntPtr ip3 = ip1 + ip2;
}

The success / failure of the above block depends on whether you are executing on 32 or 64 bit.

[tannergooding]

@jaredpar, that is where a new block type makes it consistent:

This forces checking on all platforms, it will pass on 64-bit and fail on 32-bit

checked IntPtr {
    IntPtr ip1 = int.MaxValue;
    IntPtr ip2 = int.MaxValue;
    IntPtr ip3 = ip1 + ip2;
}

However, this will pass on both 32-bit and 64-bit (whereas today, with just a regular unchecked block, it would pass on 64-bit and fail on 32-bit, if such an operator existed):

unchecked IntPtr {
    IntPtr ip1 = int.MaxValue;
    IntPtr ip2 = int.MaxValue;
    IntPtr ip3 = ip1 + ip2;
}

[svick]

@sharwell Couldn't that be solved using attributes?

Just like dynamic is an alias of object and (string name, int age) is an alias of ValueTuple<string, int>, in both cases with special semantics and preserved across assemblies using attributes, so could nint be an alias of IntPtr.

[KrzysztofCwalina]

Since the compilation of project A turns the result of MyMethod from nuint to UIntPtr, the operation in project B ends up ignoring the checked expression and calling the operator defined on UIntPtr.

@sharwell, MyMethod would return struct UIntN. The call-site (nuint value = checked(MyType.MyMethod() - 1);) would unpack the return value (convert to naitve uint) and do sub.ovf.un. The argument is that the unpacking call will be optimized out, so no perf impact.

[miloush]

Do people expect the native types to be 16-bit on platforms that are natively 16-bit?

[benaadams]

@miloush 32bit and 64bit; don't think anything is 16bit

[VSadov]

@KrzysztofCwalina: The compile time erasure as in dynamic and tuples could be worth re-considering.

(Re: @svick suggestion above)

we have considered that briefly, but dismissed thinking it would be problematic in mixed expressions like - what if you add nint and IntPtr - whose semantics to use?

Turns out IntPtr does not support binary operations with IntPtr on both sides, except for the equality, so we might not have big problems here.
Not more problems than with mixing IntN and IntPtr in the same fashion anyways.

The erasure approach would work very similarly to IntN, just without unwrap/wrap sequences in IL
(since that would be done at compile time)

The benefits are - easier interop with IntPtr.

• no wrap/unwrap in IL - just direct native int math.
• no another one pointer-like type. Not in metadata anyways.
• erased nint would be completely compatible with IntPtr. -
  Can pass nint by reference to ref IntPtr, can cast nint[] to IntPtr[] without any copying.

[tannergooding]

@VSadov, if we had that (everything is just native int under the hood and IntPtr and IntN are directly replaceable with eachother (no casting, no unwrapping, etc), then I would drop any arguments I had against the proposal to introduce a new type.

[KrzysztofCwalina]

If we go for erasure, what you the boxed type be? And what's the reflection behavior?

[VSadov]

@KrzysztofCwalina - if we go with erasure, boxed type will be IntPtr/UIntPtr accordingly, since that is the actual metadata type.
Nullable nint? will box into IntPtr as well.

I can't tell if that is a concern or actually desirable.

[VSadov]

@KrzysztofCwalina - another concern than needs to be called out is metadata interop with other compilers and in particular with old C# compilers. I am not sure how high it is on the list of priorities.

I.E. what can be done with a nint field while using old C# or other unaware compiler?

With IntN everything is clear - unaware compiler can't do much with it, other than cast to/from IntPtr

With erased nint it would depend on how metadata encoding is implemented. If we modreq the field, then an unaware compiler can't do anything with it at all.
If we use attributes, then it would look like intPtr, which is both good and bad. It is more usable, but now it runs a risk of forwards compat issues when you upgrade your compiler.

[mihailik]

The big reasoning for adding 1st class support for these types is Xamarin needs it.

OK, can we have 'xamarin scope' statement instead? In that scope [U]IntPtr acts with checked/unchecked behaviour etc.

The rest of C# use cases hardly benefit from these cool features.

c# pointer arithmetic { IntPtr x = 200; IntPtr y = 300; IntPtr z = checked(x * y); }

And leave float out, please.

The big concerns with this whole native bonanza is cognitive load for C# developers. Baking more 1st class features into the language risks turning it into PL/1, or worse - VB.

[mihailik]

Same deal as with unsafe stuff -- you can ignore the crazy piece of spec, and it sits in its kennel no barking no biting.

There is no general usage case made for these native types, so let it be compartmentalised and live in peace. If it's harder to implement this way -- even better, leave it to those needful to pay for their toys.

[benaadams]

@mihailik I'd hope it would also mean less wacky casting to deal with the correct processor length type for x86 vs x64 in a function like this https://github.com/dotnet/corefx/blob/master/src/System.Memory/src/System/SpanHelpers.byte.cs#L71-L197

[Drawaes]

Agree with Ben, accept leave float out

[nietras]

source alias approach works within the bounds of a single assembly, but causes problems when operating on results returned from a referenced assembly.

@sharwell that is perfectly fine in my view. It behaves as would be expected for IntPtr, which is the boundary interface. If you want nint behaviour you have to convert to that within the scope you are doing your operations.

I mentioned that one might constrain the use of nint/nuint to "local" scope for these reasons. I.e. nint/nuint are not allowed on public surface APIs.

removes the need for runtime changes, doesn't break back-compat, and allows users to easily follow/review the code.

@tannergooding this is why I think that type erased/aliased nint/nuint are perfect. And it avoids the pitfalls of introducing new scope types and pragmas, It is directly readable in the context it is used. Predictable in its output since it follows normal conventions etc.

leave float out, please.

I believe the whole native float thing needs to be discussed in a different issue and context. It is nothing like native int and unsigned native int and no such equivalent type exists for float. So perhaps this should be split into a native integers discussion and a float for graphics discussion, since it is really tied to that rather than float for numerical processing.

[mihailik]

I like @tannergooding's checked IntPtr block too, except for convenient-looking wording. I feel it should have more overt heading.

The upsides of 1st class native int/float are limited to calculation/integration code confined to specialist libraries. And the costs of native integer include:

• backward compatibility - libraries using new types/APIs failing on old platforms
• backward compatibility - patches and updates using new types/APIs breaking production apps running on old platforms
• backward compatibility - AOT compilation in presence of both old and new code
• stability - high risk of bugs given variety of platforms to support
• stability - 3rd party libraries needing to fork to cover for old/new scenarios, high risk of bugs introduced in those 3rd party libraries
• security - we're explicitly increasing risk of buffer overrun bugs in BCL and 3rd party code
• cognitive load - an ELEPHANT issue in itself: the spec already introduces a few tables to explan compatiblity, and even leaves holes with 'TODO' to fill later; will layman C# coder memorize that, or just treat as scary (or worse, cool) voodoo?
• misuse - given lack of guidance and vague 'use case' part of the spec, people will misuse native int
• performance hit - inflated code, class, library size, more complexity in JIT, GC, loader, interop -- all that means small local wins may well be offset back by slower platform overal

And native float comes with above plus few more:

• cognitive load - even worse: the spec is already extremely vague on float (missing conversion To/From tables, missing whole sections or even TODOs on compatibility and mapping), much easier to slip
• stability - native float behaviours are way more complex and unpredictable than integers: precision loss, NaN, 0/+0/-0, epsilon, Infinity/+Infinity/-Infinity identity; is it even possible to write generic native float code to work consistently across platform?
• stability - subtle bugs in 3rd party and BCL code due to bit representation potentially vastly different
• stability - serialization and networking incompatibility

[pentp]

Having read through this entire thread, it looks like the sanest solution is compile time erasure like dynamic (https://github.com/dotnet/corefxlab/issues/1471#issuecomment-295467537) annotated with a [NativeMathAttribute].

• This has only minimal compatibility issues (and only when using downlevel compilers).
• No runtime changes are required.
• The ToString globalization issue looks like an acceptable cost to pay and can be mitigated by implementing the overloads that take IFormatProvider parameters and using them when necessary.

@KrzysztofCwalina: Are there any other issues that this solution doesn't cover?

[jaredpar]

@pentp, @KrzysztofCwalina

Type erasure is an approachable solution from compat and language simplicity perspective. There are a couple of other minor issues that need to be addressed besides the ones listed above.

• Construction: There are a couple of subtle issues around constructing nint types that need to be addressed if we re-use IntPtr. Essentially can't go through any IntPtr ctors as we go through object ctors for dynamic instance creation. May just fall out from other rules if we force nint types to skip all IntPtr members.
• Dynamic: when nint is assigned to dynamic it will behave as IntPtr not as nint. That goes against the spirit of dynamic to some degree. Not a big issue though. Already crossed this ground a bit with tuples and deconstruction.
• Type naming: Imagine a customer has defined a type named nint today in their code. The compiler must prefer that type over the desired native int semantics. What action does the developer need to take to resolve this conflict? When there is a real type available language constructs like global:: and qualified names can help. That's not available here.

[pentp]

Construction can also be done at the IL level and respects the checked/unchecked context, no need to actually call any .ctors. One problematic area might be constant folding, but this can be worked out.

Boxing/casting to dynamic is not a problem because all the math/conversion operators are only supported on the unboxed form and unboxing must always be done explicitly to either nint or IntPtr.

IConvertible, IComparable and other interface definitions can follow the existing int/long approach (always checked).

[KrzysztofCwalina]

I think the erasure proposal needs to address boxing, reflection, and consistency issues. In the ideal world we would have Int32 with int alias, Int64 with long alias, etc. and IntN with nint alias. Where the CLR type and the alias have the same semantics. With the erasure plan, this property/consistency must hold too. Similarly, reflection and boxing needs to work exactly how it works for int/Int32 pair.

I feel like the main goal of the proposal is to have native types that behave exactly like the fixed size types (in fact I will add it as pri 0 requirement).

[pentp]

Boxing and reflection work similarly to dynamic - boxing loses the Dynamic/NativeMath attribute, reflection can detect it from the custom attribute collection. Are there any specific issues with reflection? The only thing holding us back from the ideal world of nints are the existing operators and ToString on IntPtr.

The major obstacle to exact parity with fixed size types is ToString.

[jaredpar]

@pentp

Construction can also be done at the IL level and respects the checked/unchecked context

Agree. I don't think any of the issues I listed are blockers. They're all fairly actionable. They just either deviate from the dynamic analogy or need to have some thought put into them.

[VSadov]

@KrzysztofCwalina I think in terms of type duality, mapping nint to native int/IntPtr is very similar to what happens with regular int

C# int maps to int32 in computations and to System.Int32 in metadata.
int32 is a primitive that intrinsically supports math (as in add, add.ovf IL instructions), while System.Int32 is a struct that has members, implements interfaces (when in a boxed form), works with reflection, and so on.

When you box int (or Integer in VB) and ask .GetType() you get System.Int32 . That is not inconsistent in any way - it just reveals the metadata/reflection representation of Integer.

Mapping nint to native int primitive in computations, and to its dual System.IntPtr struct in metadata/reflection seems very consistent with what we have for the regular int.

The only "funny" business here is that if you literally use System.IntPtr in your code (as opposed to nint), you get backwards-compatible math and conversion behavior.

[jaredpar]

The only "funny" business here is that if you literally use System.IntPtr in your code (as opposed to nint), you get backwards-compatible math and conversion behavior.

There is also funny business for dynamic. Consider this example an a 64 bit platform.

``` C#
unchecked {
nint value =(uint)long.MaxValue;
Use((int)value); // okay
Use((int)(dynamic)value ); // throws overflow due to using IntPtr.operator+
}

void Use(int value) => Console.WriteLine(value);
```

But I think that's mostly a minor case.

[KrzysztofCwalina]

The ToString issues is a wart, but I think we can live with it.

As to boxing, the following code needs to work (and I think it will, but want to make sure):
c# nint x = 5; object o = x; var uAlias = (nint)o; var uCLR = (IntPtr)o;

As to reflection,
typeof(nint) == typeof(IntPtr) should be true.
Invoking members on typeof(nint) and calling members on nint needs to have the same semantics.

As to consistency, both Foo(ref IntPtr x) { x=s_value; } and Foo(ref nint x) { x=s_value; } need to have the same behavior

[pentp]

@jaredpar That sample actually doesn't throw because the current op_Addition/Add method performs an unchecked add. However, allowing nint + long would be questionable at best and is disallowed in IL.

@KrzysztofCwalina Boxing and reflection would work like that. Invoking IntPtr members on nint would also work (but ToString would be a wart).
dynamic and object are interchangeable for ref parameters, nint and IntPtr would be also (this sample compiles only if s_value is IntPtr, nint, dynamic or has an user defined conversion to IntPtr/nint).

Thinking further of this ideal world of nints: C# could ignore all the existing operators and constructors on nint (including equality) and implement them in IL. For consistency with int/long there should be only a default constructor.
nint.Add/nint.Subtract and nint.ToInt32 should have at least a warning when used in checked or unchecked context respectively, but they could be entirely hidden also (together with ToInt64 and ToPointer then).

[jaredpar]

@pentp

That sample actually doesn't throw because the current op_Addition/Add method performs an unchecked add

Good catch. Had the checked portions of IntPtr mixed up in my head. Updated to a correct sample that demos the dynamic oddities.

[VSadov]

@KrzysztofCwalina yes, these scenarios are basically testing that outside of math nint and IntPtr have the same behavior.
I think all these scenarios will work as you expect.

It is the same relationship as between (int Alice, int Bob) and ValueTuple<int, int>. - As long as you do not use "Alice" and "Bob", you will not see any difference.

[VSadov]

@jaredpar - yes, dynamic behavior for IntPtr will stay the same.
Not nice, but tolerable IMO.

We could make dynamic binder to always use nint semantics, respecting "checked" context, but that would be breaking to existing code, if such exists, who knows.

[jaredpar]

@VSadov I agree it's tolerable. Mostly calling it out for completeness.

[OtherCrashOverride]

... and platform long support?
https://github.com/dotnet/coreclr/issues/2426

[KrzysztofCwalina]

I wonder if we should not treat nfloat and native long as a separate feature from nuint/nint. The former seem to be compilation concepts with no support in ECMA-335 and the later are CPU architecture concepts with existing support in ECMA-335.

Also, it seems like we are converging on a solution to nuint/nint that will be very different from what we would need to do to support nfloat (and possibly native long)

[jaredpar]

I'm okay with treating them as separate designs. But prefer we have both designs settled on before we move to implementation. Tons of shared infrastructure is going to change here, makes sense to do this together vs. separate.

[jkotas]

I agree that these should be decoupled.

Native float and long really only matter on PInvoke interop boundary and their definition is platform/OS specific. Native int is not OS specific and its usefulness goes beyond PInvoke interop.
If we wanted to have a better built-in interop support for platform float and long, I am wondering whether it should be just new UnmanagedType.NativeFloat, UnmanagedType.NativeLong values in the existing enum that the PInvoke marshalling understands, without language support.

[dsyme]

One comment from the F# perspective...

F# has always have nativeint and unativeint types mapping to IntPtr/UIntPtr (the existence of nativeint stems from OCaml which provided this type, - note F# doesn't have a native float type though).

F# implements arithmetic differently to C#, so has always provided full, correct checked and unchecked arithmetic for these native integer types.

From the F# perspective, it is frustrating that - after more or less "doing it right" all these years - we're now going to need a new, different native integer type (and basically because C# didn't originally care about these types...)

it's not a huge problem - it's good for the platform to get this sorted out. But It would be ideal from the F# perspective if under the hood nint and IntPtr were somehow exactly the same type (e.g. by using erasure in the C# compier rather than a new set of types).

That said, we can live with deprecating nativeint and unativeint in favour of nint and unint in the long term. But it somehow seems a shame to need to.

[dsyme]

@VSadov @KrzysztofCwalina Looking over the (long) thread it seems to me that you may be converging on erasing "nint" to IntPtr after all? If so, it's plausible that F# needs to do almost nothing here, since that's pretty close to what we already do.

In any case, will watch the progress

[VSadov]

@dsyme - Right. Since F# always did nativeint math intrinsically, it will not need to bother about distinction between System.IntPtr and C# nint, which is for C# back-compat.
Implementation by erasure will very likely have no actionable effect on F# whatsoever.

[jaredpar]

@VSadov, @dsyme

Implementation by erasure will very likely have no actionable effect on F# whatsoever.

Want to clarify what type of erasure we are talking about here. It is explicitly not full erasure. In full erasure it would be impossible to tell from metadata whether the developer used IntPtr or nint. That is not the case. It would be erasure + marking as we do with dynamic.

That means you get the full IDE, tooling experience that distinguishes between nint and IntPtr in cases where you should. For example in lambda bodies.

This can lead to work items for F#. Given that we will distinguish between IntPtr and nint in metadata, C# won't properly light up for F# metadata definitions which use nativeint. Those will be emitted as plain IntPtr and as such we won't see it as nativeint.

[MichalStrehovsky]

Is there a precedent for when using the language keyword to refer to a type produces a different behavior than using the underlying type? Would the type erasure approach mean that someone who puts dotnet_style_predefined_type_for_locals_parameters_members = false in their .editorconfig won't have a way to get around the bug? (I refer to the fact that the operators on IntPtr/UIntPtr got implemented as a bug, because that's what it is. Bug fixes break people, but make the product better...)

[jaredpar]

@MichalStrehovsky

Is there a precedent for when using the language keyword to refer to a type produces a different behavior than using the underlying type?

Yes. dynamic under the hood compiles to object with an attribute attached to it.

Would the type erasure approach mean that someone who puts dotnet_style_predefined_type_for_locals_parameters_members = false in their .editorconfig won't have a way to get around the bug?

This will not be a problem. From a language perspective they are fundamentally different types. This is not like the case where int and System.Int32 are aliases for the same type. Instead it's like dynamic and object. They have a relation in metadata and hence a couple of restrictions (can't overload) but they are distinct types.

[dsyme]

This can lead to work items for F#. Given that we will distinguish between IntPtr and nint in metadata, C# won't properly light up for F# metadata definitions which use nativeint. Those will be emitted as plain IntPtr and as such we won't see it as nativeint.

Yes, understood, thanks. And agreed we will need to decide if F# metadata exports F# nativeint as IntPtr or the new nint by default. Currently it is IntPtr.

For "dynamic" we just ask the F# programmer to add the attribute manually if they want the C# programmer to see it as such. We could do the same here.

[migueldeicaza]

I strongly dislike the erasing approach that is being proposed here.

[tannergooding]

@migueldeicaza could you expand on that and what your preferred mechanism would be?

[GSPP]

I do not believe that a nint type is necessary for any mainstream .NET API (Framework, libraries, applications). I see this as a way to "open the hood" for special cases. I think, for example, that AllocHGlobal(IntPtr) should not have been nint but long. This is the simpler and more "managed" way to go. There is very little downside to just computing with long in many cases.

In other words I believe the "Representing Sizes" use case is actually an anti-pattern. Why do we need a special data type just to represent the fact that some integer/pointer can only ever reach 32 bit values on some specific platform? This is a rare requirement.

I think the nint/IntPtr facility is mostly needed to declare a data layout. This is again rare but more plausible (to me) than making computations easier.

The "Interop" scenario from the document seems to be adequately addressed with IntPtr. There is just not that much .NET code doing interop and most such interop is straight forward. This would not carry the weight of even a new C#-level type. I see the "Performance" use case from the document in a similar place since computations are almost always through the pointer data types.

There is a risk that these native types might gain broad adoption and poison .NET APIs and libraries with unnecessary complexity. I had the same fears with dynamic but that turned out well.

I like Language support for checked operations very much. checked is quite useful. Provide native size floating point type also has clear value in my eyes although it might not carry its weight.

TL;DR: Can't we just make IntPtr a little nicer at the C# level? I think that's being discussed in many ways already. Saves a lot of time and concept load.

[miloush]

@GSPP The argument for no need sounds a bit contradictory to the broad adoption risk to me...

[mihailik]

@miloush look at sugary drinks, or tobacco, or Visual Basic

[OtherCrashOverride]

I think its just symptomatic of the issue that the developers that cared/needed these things have abandoned the .Net Core "dream" long ago and moved on. The message has been pretty clear that .Net core, etc are for ASP.Net.

[OtherCrashOverride]

Perhaps the more relevant question is: Is there any reason to pursue this anymore? Are there any "customers" left still interested in it?

[tannergooding]

@GSPP, One of the currently proposed approaches is to just make IntPtr nicer at the C# level via partial erasure. This would mean that there is no new nint type, it is just a language keyword (with the same semantics as var, being that if you already have a var type it will use that type instead of the keyword). It would be emitted as IntPtr under the hood with a magic attribute that indicates that it should behave with the better behavior (this is similar to how dynamic works, where it is really just object with a magic attribute).

[tannergooding]

Additionally, Interop may not be a mainstream scenario for the largest base of developers (those writing LOB apps and the like), but it is very important to those writing the frameworks (CoreFX, SharpDX, Xamarin, etc) that the afformentioned apps are built on-top of. These frameworks actually have to expose central APIs that allow users to interact with the underlying system (whether that be the OS, some graphics library, etc).

Coding these core APIs in a non architecture independent way just doesn't work. We have to be able to run APIs on both 32-bit and 64-bit hardware. This is true both for things which have been transitioned for some time (x86/x64) and for things which are still transitioning (Arm/Arm64).

Coding things to just take long is also the same logic that a lot of 32-bit programs used to be coded with (just use int, it represents the current largest platform and is big enough). Of course, that meant that thousands of applications couldn't just be recompiled and work as a native x64 application.

Pointers are sized based on the architecture and sometimes you need an integer that is treated the same way. Just on Windows, this is SIZE_T, WPARAM, LPARAM, LONG_PTR, ULONG_PTR, etc...

[mihailik]

Very good point @tannergooding, but interop scenarios do not require new language keyword, nor a new CLR type. IntPtr and * give enough expressive power.

The major concern here is that folks in interop-tinted glasses assume C# as a whole would benefit from this hair-splitting complexity. If you tinker with GC and JIT, take off your heavy boots and leave down-to-metal mindset at the door entering the ballroom of high-level C#.

A good model be the unsafe keyword: it's intentionally hard to leak unsafe in normal code. Enable compiler option. Wrap stuff in lexical scopes. Interop-tinted folks get native/interop power, but general public can remain ignorant.

If interop folks really need some new native int support (which so far described in vague hand-wavy way), it should be scoped and compiler-optioned in unsafe fashion.

[tannergooding]

@mihailik, the language design team has explicitly said they will not expose different behavior via a command line switch (I've heard @jaredpar rant plenty on why not 😉).

This isn't something that can just be ignored, because it really is a core scenario for writing framework and interop code (the basis for all other libraries).

It is true that IntPtr and * are expressive... You can use unsafe code, add a bunch of casts, do pointer arithmetic, and cast back to get things "working". But this also means that some of this central code isn't optimal.

For example, because of all these casts and operations that are getting IL emitted for them, the JIT has to work harder to produce optimal code, rather than just emitting code and following code paths for the existing IL instructions it has (which were designed just for this purpose). Given that this code can be called potentially hundreds of times a second (like the interop code required for UI code is), that really isn't a good place to be in.

So given that this needs to be done (or is considered to be needed by enough people with enough influence), the big argument so far (and the argument on why to do it this way or the other way) has been back-compat and working with existing APIs.

We can't just expose a switch because that makes it difficult to understand the behavior of the program just by looking at it (this is one of the issues with the checked command line switch).

We can't just make IntPtr do the new behavior because it means recompiled code will secretly take on new semantics, possibly breaking existing programs.

This leaves (essentially) two options: Either introduce a new type or do something else (currently proposed to be partial type erasure). There are benefits and drawbacks to each (mostly around existing APIs in Xamarin vs existing APIs in CoreFX), but those have been discussed and argued about enough in other threads.

I also don't find it (partial-erasure) to be very complex at all. With it, we have an 'nint' keyword that causes math operations to work and we have a 'IntPtr' type where they don't. You can convert from 'nint' to 'IntPtr' (and vice-versa). To most users, they are as distinct as dynamic and object. But to users in the know, they are the same type.

[SamuelEnglard]

@tannergooding @mihailik may I suggest a compromise: nint (and the like) only existing in unsafe context?

[mihailik]

@SamuelEnglard @tannergooding brilliant idea!

Use-cases of native int and unsafe significantly overlap -- simple to conflate the two for the initial stage implementation.

That way nobody can step in it by mistake, and yet it's available for experiments. Good Xamaritans can adopt it for some code, and we'll have serious material argument for expanding the usage in v.next++

[pentp]

There is nothing "unsafe" about nint. The stated purpose of this proposal is to provide native sized integers with functionality and ease of use on par with int/uint. There are many use cases for this, only some of which are related to interop.

Most of the discussion here has been around finding the best solution with minimal downsides and the consensus we've arrived at is to polish out the existing IntPtr type with native math support in C#. If the only arguments against it are "cognitive load" and an unfounded fear of it being somehow unsafe, then it looks like a pretty good solution.

[Drawaes]

If there are "many use cases other than interop". I would be very interested to hear them so I could better understand. Also the floating point seems to has stopped being discussed can I take that to mean it has been dropped?

[mihailik]

The point is gradual introduction of native int.

Allow evaluation in interop scenarios, at no risk to general C# use cases. If (when?) this reveals material value and actual cases -- native int can leave the unsafe jail.

The interop-tinted bias conflicts with LOB-minded bias, so let's gather real world usage at low risk, and sidestep bias battle. It's a win-win solution.

[SamuelEnglard]

@pentp while true that there is nothing "unsafe" about nint, in most if not all cases where it will be used the unsafe flag will already be flip for the compiler and will most likely be used nearby. As @mihailik said, the overlap between the two use cases is high enough to prevent LOB developers from coming across and using the type unnecessarily or incorrectly.

On top of which in C# there is only so much "unsafe" you can really do (unless you're calling native methods to do it for you).

[tannergooding]

@mihailik, I'm interested in how you think native int is a risky type to LOB apps?

The only risk associated with it (that I am aware of) is that it can be 32-bits on computer A and 64-bits on computer B. This requires some minimal understanding that some operations will cause underflow/overflow on A, but will not always cause the same on B.

[SamuelEnglard]

Two quick reasons I would not want random developers to see nint

1. Intellisense bloat: autocomplete is how lots of developers find a type when they can't remember the name. Last thing I want is to make the list longer!
2. Forcing them to cast: IntPtr (and therefore nint) cannot be implicitly cast to int or long so using it means a cast every time you want to pass it to a "normal" API

[tannergooding]

1. IntelliSense will continue growing either way, that's why the Roslyn IDE team is making improvements to make locating the relevant types easier.

2. Casting IntPtr to int or long is generally bad practice in the first place (which is why it is explicit). Ideally, it should be treated as its own integer type and should be handled that way end to end. Casting to int is only okay on 32-bit machines, on a 64-bit machine you risk loss of data. Casting to long is okay on all existing architectures we support, but it still leaves future compatibility concerns. Users doing things like casting IntPtr (or raw pointers) to int (or just using int in the first place) is exactly why we still have so many apps that are still natively 32-bit today (because doing such things breaks cross-compilation).

[tannergooding]

To elaborate on 2, that is one of the reasons why creating the nint type is required. There are a lot of native APIs today (especially on iOS, Android, and Mac -- but plenty on Windows and Linux as well) that take things like size_t. In order to work effectively in the framework on both 32-bit and 64-bit architectures, we need a "safe" platform-sized integer type that can be used end-to-end.

If we just say 'use long' everywhere, then we will either break in the future (if we ever get something other than 64-bit hardware) or, at the very least, we will be incredibly inefficient on 32-bit OS (requiring twice as much space for every native int backed parameter and doing any operation will require double the cycles at least (its significantly more for things like multiply and divide).

[SamuelEnglard]

I agree with all your points about replying to 2 BUT those are reasons to have the nint, not reasons the average developer should see it.

However, you bringing up iOS and Android is an interesting point. I don't know this, but I'm curious if it's more common to need to have an API return a native sized integer and do math on it. If so, then yes, only having nint inside unsafe makes no sense anymore

[tannergooding]

That is one of driving factors of this proposal: iOS/Android/Mac APIs which deal with nint (that is deal with native sized integers, not pointers/opaque types -- see https://developer.xamarin.com/guides/cross-platform/macios/nativetypes/ for more details).

[CyrusNajmabadi]

I'm a bit lost at this point what the compelling case is for this to be in the C# language proper.

Is there a reason we can't just have a type called System.NInt, and provide operators on it? Why does the language need to be involved? Thanks!

[tannergooding]

@CyrusNajmabadi, you should probably get the explanation from @jaredpar (just because he is much better at explaining it then I am)....

However, in my opinion, the primary reason is that it is the easiest option overall and requires the least code churn across CoreFX/CoreCLR/Roslyn.

The runtime currently supports IntPtr as a runtime primitive, so if we were to go the route of adding a new type, it means that the runtime would possibly need updating (for performance, interop, etc).

There are tons of existing APIs that use System.IntPtr to represent a platform-sized type (both integers and pointers/handles/opaque types). A new type means that you can't just work with any existing APIs (that have been created over the past 17 years) without first doing casting/etc (this results in extra IL that the runtime has to JIT and optimize).

There are other .NET languages (F# for example) that already treat System.IntPtr as platform-sized integers. So a new type means they have to be updated to handle something which the runtime has supported since day 1 (which, in my opinion, is a silly thing to do just because a more widely used language decided not to implement the feature before .NET 4.0 came out).

[jaredpar]

@CyrusNajmabadi

Is there a reason we can't just have a type called System.NInt, and provide operators on it?

This is actually exactly what Xamarin does today. The experience is functional but it's definitely not first class. The corner cases bleed through and can make the experience unpleasant.

Why does the language need to be involved?

The original intent is to make working with native int as natural as working with any other primitive arithmetic type: int, long, etc ... Essentially make the type first class. That included the ability to:

1. Use as constants including sub-features like constant folding.
2. Use as default parameter values.
3. Support checked / unchecked contexts.

These require language support to implement.

[CyrusNajmabadi]

Could you clarify that a bit @jaredpar . For example, it's not clear to me how constants would work with a type that is effectively platform/runtime dependent. The compiler doesn't know if it will be 32bit or 64bit, so it's not clear how it could effectively represent any constant value. Or would we require that any constants for nint be less than 32bits, and all operations on it would not be allowed to overflow when evaluating?

--

Default parameters seems pretty low pri for me. But i can see how wanting checked/unchecked semantics might be nice for someone to have.

[sharwell]

@CyrusNajmabadi For me, the driving requirement for language support is being able to emit proper IL sequences for these types, as opposed to calls. Most instructions that work with integers are defined to work with native-sized integers, but are not exposed to people coding in C#. Even if the JIT were able to inline all these calls, that approach increases the cost of JIT itself in order to support executing the code efficiently.

[CyrusNajmabadi]

IL isn't something the language is concerned with though. It can be something the compiler/jit do as necessary. IL is just an intermediate representation, and just an implementation detail of the current C# compiler. But all of that is flexible without needing anything done at the language level.

Jared's cases are actual cases that would necessitate language level support. To me they seem rather marginal. But i'm not necessarily coding to these APIs day in and out, so they may actually be very important.

--

It would be helpful to perhaps get examples of the sorts of APIs we're talking about today, and how they would look with native language support, instead of just exposing through some type and some set of operators over those types. A lot of this discussion seems to just be about nitty gritty low level details, but i feel lost as to what teh user experience will be here, and why it would mandate direct language support.

Thanks!

[sharwell]

@CyrusNajmabadi My comment was referring to two things in particular:

1. Not having to define the operators in metadata for the IntPtr and UIntPtr types (the same way the spec already handles the operators for other integer types)
2. checked/unchecked behavior

[CyrusNajmabadi]

Not having to define the operators in metadata for the IntPtr and UIntPtr types

This just doesn't seem valuable enough to me from a language perspective. I'd much rather just define operators in metadata than try to change the language. To me the language changes happen when either there is great value to them being in the language proper, and you can't already get close enough with other available means.

--

But again, i don't abide by "the driving requirement for language support is being able to emit proper IL sequences for these types, as opposed to calls." "calls" are perfectly fine with me. If tehre's actually an issue (which i'm skeptical about to begin with), it could be addressed in the runtimes used on the platforms where there would be higher desire for 'nint'.

--

In other words, it's really unclear to me that adjusting the language is the way to go here. I'd much rather keep the language as is, add APIs/runtime support as necessary, and see if that was sufficient. You would lose checked/unchecked it seems. But i'm again skeptical that that's actually of significance for those that want nint support. And even if it was that significant, it seems so marginal it still doesn't feel like something i would think would be worth elevating to a language feature.

[jaredpar]

@CyrusNajmabadi

For example, it's not clear to me how constants would work with a type that is effectively platform/runtime dependent. The compiler doesn't know if it will be 32bit or 64bit, so it's not clear how it could effectively represent any constant value. Or would we require that any constants for nint be less than 32bits, and all operations on it would not be allowed to overflow when evaluating?

This is essentially the model I had in mind: restrict nint constants to 32 bit as it's the only values we know to be safe.

In other words, it's really unclear to me that adjusting the language is the way to go here.

Should we do the work is definitely a discussion to have. For the purpose of this particular Github issue I've been trying to keep my focus on the "assuming the need is there, what is the best way to approach the design".

[sharwell]

@CyrusNajmabadi I wasn't trying to debate the points again; already did that in earlier comments. I was just stating what the driving factors were for this feature for me since you asked.

Should we do the work is definitely a discussion to have. For the purpose of this particular Github issue I've been trying to keep my focus on the "assuming the need is there, what is the best way to approach the design".

Me too, for sure.

[CyrusNajmabadi]

@jaredpar

For the purpose of this particular Github issue I've been trying to keep my focus on the "assuming the need is there, what is the best way to approach the design".

That's fair.

I would like some help with the "Assuming the need is there" part. Actual examples of where people are running into this would be helpful. Along with examples of what we'd want to see in the language proper. It would help me gauge how valuable this would be to be made at that level and what marginal utility would then be gained over a non-language approach to this problem space.

[pentp]

One example that has been linked to here before is SpanHelper.IndexOf. There are many casts and other workarounds there that would be unnecessary with nint.

One interop related example - avail_out/avail_in are of type UIntPtr (SIZE_T), but using them needs a bunch of casts.

[CyrusNajmabadi]

Note: I'm not asking for code that could benefit from a nint type with appropriate operators. I'm specifically asking for code that would require such a feature to exist at the language level.

In other words, if you had the following provided by the runtime:

c# struct Nint { // whole bunch of operators .... }

Would you need direct language support, or would that be sufficient?

[pentp]

It needs language and runtime support because it's performance critical and/or interop code. Obviously the current code works, but the whole point of this proposal is to make such code less painful to write and easier to read also.
IntPtr has all the runtime support already in place, a new type would require significant work on multiple runtime components (JIT, pinvoke, COM-interop). Any approach would need language level support if it's going to improve the current situation.

[mihailik]

Here's an approach that runs at full native speed, but does not involve language change:

Implement 2 types below in C#, but make JIT swap operator calls with native code.

```c#
struct CheckedNativeInt
{
public IntPtr Value;
public static CheckedNativeInt operator+(CheckedNativeInt x1, CheckedNativeInt x2) { /* ... / }
}
struct UncheckedNativeInt
{
public IntPtr Value;
public static CheckedNativeInt operator+(CheckedNativeInt x1, CheckedNativeInt x2) { / ... */ }
}

```

Note that Xamarin folks, have been doing JIT-side operator substitution 9 years ago:

Mono's SIMD Support: Making Mono safe for Gaming (2008)

Also note that actual code samples above using native int operate in unsafe context. Both of them!

[tannergooding]

@jaredpar

This is essentially the model I had in mind: restrict nint constants to 32 bit as it's the only values we know to be safe.

This is indeed the only thing that makes sense and the only thing that the CLR currently allows.

The load constant (ldc.*) instructions are used to load constants of type int32,
int64, float32, or float64. Native size constants (type native int) shall be created by
conversion from int32 (conversion from int64 would not be portable) using conv.i or conv.u.

[mihailik]

The risk of introducing variable-bit numbers into LOB code written under fixed-bitness assumptions are well explored during 16/32 and 32/64 transition:

Corruption and loss of data, overflow bugs, serialization bugs.

And in strict terms, the problem is as follows. LOB and other high-level code is operating domain-oriented logic over domain-oriented state. Neither of those bears any relation to 32/64 bitness. Any occurence of native int in LOB/high-level code is therefore a bug: mismatch between the storage type and the value stored.

[tannergooding]

@CyrusNajmabadi, that would not support any of the three items @jaredpar listed as the things that do require language support:

1. Use as constants including sub-features like constant folding.
2. Use as default parameter values.
3. Support checked / unchecked contexts.

Use as constants could partially work if we had a larger language feature which allowed constants for any type (this would require runtime support). Constant folding may or may not work on these other types (constant folding likely requires some intricate knowledge about the type in order to be done properly).

Use as default parameters could also be done as an extension on the above. I would assume this (and the aforementioned) would need to be restricted to "blittable" types.

So, it really comes done to three, which we have no good way of supporting, in any fashion, without language support. We can't use 'checked'/'unchecked' with user defined operators (which is what these would be if implemented as a new System.nint type or even implemented as operators on the existing System.IntPtr type).

We know that IntPtr is a core interop type (for obvious reasons). So, the question that @CyrusNajmabadi wants answered (correct me if I'm wrong) is whether or not providing language support for checked/unchecked semantics on IntPtr is worth the time required to design and implement said feature.

[mihailik]

@tannergooding checked/unchecked semantic can be trivially achieved by having 2 separate types (see CheckedNativeInt and UncheckedNativeInt above).

Good points on constant features though. It would be nice to extend constant-related language features to blittable structs! One way to do it is only allow constants in form of new MyStruct { FieldX = 1, FieldY = 20 }. You can't run constructors or setters at compile time, but you can assign fields.

[mihailik]

@tannergooding please note that adding new feature isn't costed by design and implementation effort. Otherwise we'd have Matt operator already.

The major cost here is risk of volatile bugs in LOB code: works on my machine fails in production.
And of course the usual cost: maintenance, blocking other future work in the same area.

[tannergooding]

@mihailik

Any occurence of native int in LOB/high-level code is therefore a bug

I disagree with this statement completely. It certainly increases the chances of a bug when interacting with certain types of data (such as most data that is actually stored in a database). However, the core operating system and framework APIs have to work with native int and it is either a bug or a future-compat issue to stop using native int at any point in the stack (it is a bug to use int because it doesn't work on 64-bit and it is a future-compat issue to use long because we may have machines with a different bitness in the future).

There are literally thousands of examples of how not having native int end-to-end is a bug. If you just go skimming through various OS APIs you will see where they had to introduce new types or change things to make them keep working (what used to be a pointer is now an index into a table so that the API works on 64-bit).

Additionally, there are plenty of LOB apps that function perfectly when working with native int types. The APIs for Mac/iOS are completely designed around this (see the NSInteger type). Xamarin is also designed around their System.nint type because of this.

[CyrusNajmabadi]

It needs language and runtime support because it's performance critical

This does not follow. You can have performance critical code generated without any language support. I can define things as a struct + operators, and the jit could literally erase that all away into single CPU operations.

and/or interop code.

I see no reason why it being interop code means that we need direct language support for it.

[CyrusNajmabadi]

that would not support any of the three items @jaredpar listed as the things that do require language support:

Again, i'm straining to find value in the things that jared mentioned.

Use as constants including sub-features like constant folding.
Use as default parameter values.

I can easily live without these. I don't see why those are so necessary. What are the cases whereby it would be so critical to have this at the language level?

Support checked / unchecked contexts.

This is a bit more interesting, but i'm still straining to see why i should care about this? That's why i asked for non-abstract ideas, but instead actual code samples that demonstrate why the above is both extremely important, and it's necessary to have at a language level (as opposed to say, the CheckedNativeInt and UncheckedNativeInt approach outlined above).

--

I'm not saying people don't want this. Or that it wouldn't be useful. I'm arguing that the marginal benefit of having this be a language feature seems so small as to not be worth it. If all i'm getting is the ability to use this sort of thing as a constant/default parameter (i.e. a constant), then that just doesn't seem worth it.

[CyrusNajmabadi]

There are literally thousands of examples of how not having native int end-to-end is a bug

I have no problem with the idea of a native-int. It makes total sense to me. What i question is why such a thing needs to be surfaced at the language level in a special way. Why is it not sufficient to just have a framework type that the framework treats appropriately as a native int?

--

Let's look at the examples of where this has been listed as an issue:

1. Default parameter values.

This is highly uncompelling to me. If you need this, you can write:

c# using nint = System.Nint; void Foo() => Foo(SomeVal); void Foo(nint n) => ...

The runtimes will happily inline all of this.

1. Constant folding

I also find myself not feeling like this is important enough to warrant language inclusion. If this really is something that benefits your program, i don't mind just saying "just compute the constant value yourself and embed it directly".

In other words, the use cases are not interesting enough or common enough to say "this must be in the language itself"

1. Checked/unchecked.

I'd like to understand the cases where this is so important that it warrants direct language support and could not be provided just through special types that that the runtime compiles down directly to support this.

[CyrusNajmabadi]

So, it really comes down to three, which we have no good way of supporting, in any fashion, without language support

But are these three things actually important? I was really hoping for a list where i would say "oh, ok yeah... that's pretty darn important. i can see why we'd def need this in the language". Instead, the list seems pretty 'meh'.

As i mentioned before, can people please link to actual code cases that demonstrate why this would be necessary as a language feature. So far, all the examples are those that could be done using a framework type without any language support at all.

[tannergooding]

@CyrusNajmabadi

I'd like to understand the cases where this is so important that it warrants direct language support and could not be provided just through special types that that the runtime compiles down directly to support this.

You could reverse this and ask why the runtime should have to add extra support for a new special type that provides this support when they already (since v1.0) have provided System.IntPtr specifically for this case. Other languages (such as F#) are already using System.IntPtr in the desired manner (by exposing the appropriate runtime defined operators).

Us doing anything other than supporting the type which the runtime already defines puts additional pressure on the runtime to ensure this is performant (essentially making them have to handle two code-paths as IntPtr and new type now need to do the same special handling). It also means that users wanting to consume existing APIs have to perform casts to get things working. This also means that certain APIs (ref IntPtr, IntPtr[], etc) will not be useable outside of .NETCore (.NETCore only works because we provide a special Unsafe class that allows you to do things like convert ref T to ref X, but doing so is also very unsafe).

Also, working on getting together some good solid examples, will have it up later tonight most likely

[svick]

It seems to me that the issue here is that one side says "it doesn't have to be a language change, it can be just a library+runtime change", while the other says "it doesn't have to be a runtime change, it can be just a language change".

My question is: Is it clear which of the two approaches would be cheaper?

I'm asking because if the "language change" approach was cheaper, then I think it's clearly preferable, since there are some arguments in favor of language change, and the "it can be just a library+runtime change" argument is not very persuasive, if that approach was actually more expensive.

[CyrusNajmabadi]

You could reverse this and ask why the runtime should have to add extra support for a new special type that provides this support when they already (since v1.0) have provided System.IntPtr specifically for this case. Other languages (such as F#) are already using System.IntPtr in the desired manner (by exposing the appropriate runtime defined operators).

Form the conversation it sounded like it was because the IntPtr semantics are nto what people wanted. So it would be beneficial to have a new type that did accurately represent the semantics that people wanted.

But yes, i would absolutely reverse this. I see no problem makign the question about that.

Us doing anything other than supporting the type which the runtime already defines puts additional pressure on the runtime to ensure this is performant

This is not true, and i obviously haven't been making myself clear. The runtime can simply define the types and from the language's perspective it will work just as all types do.

The compiler is free to do whatever it wants, including just compiling these operations down to specific IL instructions. What the compiler does is not the purview of the language as long as the language abides by compiler rules.

--

Note: this is not hte first time that the compiler takes knowledge it has about the core types and optimzes things around it. For example, if i'm not mistaken, if you have a readonly primitive type (like "readonly int", and you call something like ".ToString" on it, the compiler is smart enough to know not to copy that before calling ToString. It knows it doesn't need to do that because System.Int32 is immutable and non of the methods on it actually mutate it. It can't make this determination about other value types, and so it must copy.

--

So, again, i'm asking why the language (i.e the C# language spec) needs to have anything done here to support this. So far, i can see 4 reasons for it (the three Jared mentioned and the additional one is being able to say 'nint' without defining a using alias at the top of your file). That doesn't seem worthwhile to me given that there are approaches it feels like we could follow that give a 'good enough' feature without needing any language cahnges at all.

And, if the way i've talked about is not 'good enough' then please demonstrate that with actual code cases that will help me understand why we need this in the 'language proper'. As i've mentioned in the past, i'm swayed by actual examples. Continually just going back to the abstract asks isn't convincing to me as i can't see why those are important enough to outweigh the types of solutions i've outlined myself for this problem space.

[CyrusNajmabadi]

It seems to me that the issue here is that one side says "it doesn't have to be a language change, it can be just a library+runtime change", while the other says "it doesn't have to be a runtime change, it can be just a language change".

Just to clarify: i'm trying to find if it needs to be a language change, or could it just be a compiler change. i.e. could we use existing things we've already shipped and have support for (i.e. Types + Operations etc.) and just have the compiler know that it can more efficiently by emitting direct IL.

[MichalStrehovsky]

How much thought was put into the "obvious, but breaking" solution for this: have the C# compiler simply ignore the operators on IntPtr/UIntPtr when compiling for language version X or higher? Was it dismissed just because it might break people?

Do we have any data this would actually break people (not in theory, but actual behavior changes)? How bad would it be?

ApisOf.net seems to indicate people really don't do much arithmetic with these in C# (op_Subtraction has a whooping 0.1% usage, op_Addition comes at 2.5%) - we could review the usage there to see a real world sample. I would bet that most of these actually do pointer arithmetic - and while the checked semantic could save you, code that relies on integer overflows to detect bad pointer manipulation is in a pretty bad place anyway. You can't use IntPtr as a good numeric data type unless you can do other stuff with it (divide, multiply, bit shift, etc.). People haven't been using IntPtr as such in C# much. They can use it in F# and other languages though - using the same type would make interop with F# much more natural.

[CyrusNajmabadi]

Talked to tanner offline. He pointed out another interesting case that would not be handled by my approach. Specifically that if you had an IntPtr[] you would not be able to move that to a Nint[] trivially. Definitely interesting, though i'm not sure if it would be important in practice.

[mihailik]

There are literally thousands of examples of how not having native int end-to-end is a bug

I have no problem with the idea of a native-int. It makes total sense to me.

Pointer-sized integeers make no sense in high-level code, LOB code.

Large majority of C# out there operates on entities like orders, holdings, shipments, shopping trolleys, money, weight etc. All that massive chunk of code has nothing to do with pointer-sized integers.

It looks unprofessional that such serious decisions with long-term consequences are discussed off-the-cuff, with imaginary 'thousands of examples' (did they have sheep shortages last night??) and open contempt to LOB code needs.

Again, if low-level code needs pointer-sized integers that badly, introduce it in unsafe context first and assess the viability before unloading the burden onto the general high-level codebase.

[jnm2]

All that massive chunk of code has nothing to do with pointer-sized integers.

Nor does any LOB code have anything to do with 32-bit-sized integers or 64-bit-sized integers. When is size relevant, so long as the implementation of the logic is correct?

[tannergooding]

@mihailik, it is only invalid in cases where you dealing with the backing model. Using native int can be completely valid when dealing with the view (most UI stacks have to deal with native sized integers at some point).

For example, on Windows, every message that is processed by the message pump carries two pointer-sized integers (wParam and lParam). In some cases these represent pointers to data, in some cases they represent actual platform sized integers.

For someone working with the actual UI components, they will need to continue working with native int the entire time. This will never be visible to the model, but it is also not unsafe (and nor should it be restricted to being unsafe).

[jaredpar]

@MichalStrehovsky

How much thought was put into the "obvious, but breaking" solution for this: have the C# compiler simply ignore the operators on IntPtr/UIntPtr when compiling for language version X or higher?

Considerable. I've easily spent as much time discussing this as any other potential design.

Was it dismissed just because it might break people?

Dismissed is not quite the word I would use. More that of all the options re-using IntPtr by the same name is the least palatable one. The downsides are likely strong enough that it would disqualify it as a potential implementation strategy.

Do we have any data this would actually break people (not in theory, but actual behavior changes)?

Taking a step back here. It's important to understand the position of the compiler team when it comes to compat and that is simply compat is king. There is no feature we take more seriously than compat and we go to great lengths to maintain it. Not because it's fun (it's actually quite maddening at times). But because it's a feature that our customers demand of us.

The C# customer base is huge as is their code base. Even a compat change that effects 2.5% means that you're potentially impacting tens of thousands of users. That's thousands of users who upgrade to a new Visual Studio and suddenly can't build their applications (and hence won't upgrade). It's bad enough when we unintentionally break customers in this way but doing it deliberately is a non-starter. The few times the language has deliberately broken compat have been done due to fairly extreme circumstances.

For IntPtr is easy to outline very plausible use cases which would break if we changed the API in the manner described in this proposal. Worst is that the compat break wouldn't be a compile time error (bad but at least customers know they're broken) but instead random runtime behavior changes. That's pretty much the worst case because customers often don't find out until they deploy that we rather silently broke them. This combined with the age of the existing behavior (around since C# 1.0) means the compat risk here is extremely high. Much higher than our tolerance levels.

code that relies on integer overflows to detect bad pointer manipulation is in a pretty bad place anyway

That's not a good justification for breaking compat. This is well-defined behavior that customers have been able to rely on since C# 1.0. Just because it's not ideal, doesn't mean customers won't depend on it.

[tannergooding]

@jaredpar

since C# 1.0

That isn't quite true. The framework operators were actually only added to the IntPtr type in .NET 4.0, so it is actually "relatively" new (7 years of behavior, rather than 17). If we had gotten to this proposal in 2009, we could have just done the operators on IntPtr without worrying about any breakages (not that this changes the stance 😄)

[mihailik]

@jnm2 that is precisely what I mean by 'open contempt to LOB code'

All that massive chunk of code has nothing to do with pointer-sized integers.

Nor does any LOB code have anything to do with 32-bit-sized integers or 64-bit-sized integers.
When is size relevant, so long as the implementation of the logic is correct?

Trivial question asked in flippant way: shows how little effort was put into LOB scenario evaluation.

Historical 16/32 and 32/64 migrations demonstrated the risk: unintended overflow, API mismatch, file/network serialization mismatch. Take any nontrivial LOB app and find-replace int->long, you'll see plenty of trouble.

@tannergooding your strawmen are unhelpful. Read the WinForms implementation code, there is a deliberate effort to insulate pointer-sized data from higher-level API. IntPtr is superior for this scenario: pointer-sized integers must not leak to higher levels.

Most UI frameworks using pointer-sized integers -- can there be any clearer sign of bias?

WPF and UWP uses double, WinForms uses int, HTML has number, Android uses int or higher-level units.

When you say 'most' is it 'most gadgets in my house', 'most stuff I work on' or is it in any way related to the people and things out there in real world?

[jnm2]

@mihailik I'm trying to get to the root of the issue without being flippant. I write nontrivial LOB code. I intentionally insulate the domain logic from dependence on a particular integer size, even floating point quirks. Application infrastructure serializes ints in a size-independent fashion. Changing int to long would simply cause our applications to run faster on x64 machines. I could be wrong but I'm pretty sure that if your LOB code takes a dependency on a certain word size, it's poor design.

[tannergooding]

Win32 uses WPARAM, LPARAM, LRESULT, INT_PTR, UINT_PTR, LONG_PTR, etc..
Mac/iOS uses NSInteger, CGFloat, etc...

All of those types, used throughout the various APIs (both UI and non-UI) are platform-sized integers (or platform-sized floats in the case of CGFloat).

[mihailik]

@tannergooding at least 75% of those types illustrate the success of current status quo.

Win32 API use in C# works absolutely fine with IntPtr and pointers; 17 years of history show that pointer-sized integers add no value and never will, as Win32 isn't changing much.

[mihailik]

If it's all about Apple and Xamarin, would anybody please clarify: where exactly is the savings?

Does Xamarin animate things by calling into iOS 60 times per second, and loses on marshalling?
How do you animate angles, with whole number of degrees?
If it's not invoked at 60FPS, how much is the cost of struct/operator-based solution over say 100K cycles?

[rolfbjarne]

@CyrusNajmabadi: more things you can't do without language support (all constant related):

1. you can't declare nint constants

2. you can't use nint in attributes without language support.

error CS0181: Attribute constructor parameter 'value' has type 'nint', which is not a valid attribute parameter type

1. you can't declare enums whose underlying type is nint (ecma-335 allows the underlying type to be native int, I just checked).

Unfortunately all of these are limited in their usefulness because their constant values can only be 32-bit (without runtime changes).

[mihailik]

@rolfbjarne you've got two of a "5"

To your actual point, full-width 64-bit constant support for this type would take more than just "runtime changes". Without expensive hardware modifications such values may require hard physical labor to fit into x86 CPU registers.

[rolfbjarne]

@mihailik fixed the double 5 :smile: Of the top of my head I can think of a couple solutions to the 64-bit constant support:

a. Encode two constants in the IL, one for 32-bit and one for 64-bit.
b. Chop off the higher bits on 32-bit platforms (maintaining sign).

It's not an unsolvable problem, and smarter heads than mine might even come up with better solutions. I doubt it's worth it though.

[jnm2]

@rolfbjarne Yeah, but what's the use case for that?

[rolfbjarne]

@jnm2 CyrusNajmabadi didn't ask for use cases, just constructs that required language support, and couldn't be implemented using normal structs.

For a use case: Xamarin has been mentioned before, to bind Apple API. Apple defines enums using NSInteger (which is 32-bit long on 32-bit platforms, etc), and these enums have may have different values depending on 32-bit and 64-bit code (NSIntegerMax for instance).

[mihailik]

NSIntegerMax doesn't look like an enum.

[tannergooding]

If you really want 32 vs 64-bit constants, you can define actual int constants and cast to IntPtr (which is what the runtime dictates you do for 32-bit constants anyways).

[rolfbjarne]

@mihailik it's an enum value, not an enum.

The equivalent C# enum would be:

enum E : nint {
    Value = nint.MaxValue,
}

[mihailik]

It has nothing to do with enums whatsoever.

Pushing square pegs through round holes -- that's what it is. Quantum thinking: constants that actually vary, machine words bigger on the inside, variable-length encoding free of variable-length problems.

This feature has not been justified in concrete examples, nor costs were estimated properly. Putting it in unsafe scoped jail is the least measure to limit the risks.

C# compiler team has been famously rigorous in weeding out non-features long before they surface in language. Please do not lower the standard for the sake of the whole community.

[Drawaes]

Again with the Apple use case. Apple itself is going to drop 32bit. If it does at the announcement/conference next week can we put that to bed. Seems insane to add something just for that use case. I am yet to hear of another compelling use case so far that doesn't do just fine with intptr. As for the floats that is a busted flush in my view.

[tannergooding]

@mihailik

I very much doubt that IntPtr.MaxValue would ever be exposed as a constant. For one, it isn't portable, so it isn't useable on AnyCPU. Additionally, the CLR spec doesn't technically support 64-bit IntPtr constants (all references to them indicate that IntPtr constants are really just 32-bit constants that undergo an explicit conv call -- this is what the runtime supports for IntPtr sized enums, for example).

As such, things like IntPtr.MaxValue would get exposed either as properties or as static readonly variables (see my proposal for this here: https://github.com/dotnet/corefx/issues/20256).

Additionally, unsafe isn't some "jail" just to put features you don't want people using it. It is specifically for things which are unsafe to use (specifically this should be anything that breaks memory or type safety). The feature does neither of those things. It is just dealing with integers, which to the program executing have an explicit size. There are some "gotchas" when dealing with things like serialization or marshalling, but no more so than when dealing with any other kind of data.

Every data storage format has a contract in how the data is stored and how it must be read. For example, a large portion of data is stored in little-endian format. Any big-endian system needing to access that data needs to know that it needs to perform a byte-swap before actually touching the data. Likewise, with IntPtr, it is really no different than serializing any other variable sized piece of data (arrays, strings, structs like BigInteger, etc...). You need to store the size of the data and then store the data itself.

If you are really that concerned about it, then you can write an analyzer that prevents you from using it in your codebase.

[Drawaes]

The intention might have been that for unsafe, but it definitely acts as a barrier for LOB devs, I look at a lot of code and help other teams, and there is a real fear of turning unsafe on

[mihailik]

@tannergooding code analyser as an entry barrier to writing simple high-level code??

No jokes, if you need to understand LOB space, best way is to spend 17 years doing it. Or ask somebody who did.

Same goes for deserializing 64-bit values into 32-bit runtime, just ask someone competent for an explanation.

[pentp]

@mihailik what's your problem? There is no reasonable excuse for acting this impolite, even if you personally fear this small addition to the C# language or just hate the world.

I'm 100% sure most people in this discussion are competent programmers and the C# language design team has some very talented people thinking about these kinds of language changes. The feature is useful and the principal downside is additional complexity.

So please leave your ad hominem arguments and unsubstantiated fear out of this discussion.

[mihailik]

@pentp not sure what problems you're talking about, or what hate/fear. Can you stick to the topic? Please.

Marshalling of platform-dependent pointer-sized types is fundamentally unsafe. That is, extracting 64-bit value from stored location into runtime 32-bit location leads to unavoidable data corruption.

There's no shame in missing this trivial fact, we all have different levels of competency and I for once am happy to spell such things out, as I did earlier.

Same way I have spelled out risks of this feature to the wider community. @pentp please do not bring personal feelings into this, it's about material measurable risks to large stable and expensive codebases out there.

As an active member of C# community for decades, and a long-term collaborator with Microsoft on CLR platform and other projects, I expect to see better moderation and more professional discussion driven by competent seasoned staff.

When reasonable concerns are raised, they should be seriously taken into account and not brushed off. Treatment like 'if you fear for your code' or patronising contempt to legitimate feedback is very counterproductive to the ecosystem. I understand the proponents of this feature acting like this without Microsoft's mandate, but I expect the vendor and sponsor to put more effort in setting the rational, measured, evidence-based tone.

Features are judged not on whether they are feared or desired, but on their merit. And showing how exactly a change makes a visible measurable improvement. Let's all remember that.

[jnm2]

There is no reasonable excuse for acting this impolite

Seconded.

@mihailik

Marshalling of platform-dependent pointer-sized types is fundamentally unsafe. That is, extracting 64-bit value from stored location into runtime 32-bit location leads to unavoidable data corruption.

And for those reasons, no one does this. Having nint will not cause people to start doing this.

[Drawaes]

Agreed on the let's avoid personal insults.

I will tell you this, I have a lot of experience with the corporate .net world. Unsafe is almost never used, Intptr sounds dangerous to as does interop. When any of those are needed usually "specialists take care of it" but something that is a keyword in the language right along with int and unit... well I will just give that a shot.

My problem here is you are proving a foot gun and making it seem very normal to carry it around d in your pocket. I would like to see some barrier to entry.

Also I will say it again, just in case repetition works there is no good justification for native floats that I have seen other than a soon to be defunct Apple api.

[miloush]

I do see a conceptual difference between IntPtr and native-sized integers (or floats). You would use the pointers to point to something in the memory and native-sized numeric types to do computations. You pass things to the underlying OS in IntPtrs but let the CPU do arithmetics in nints or nfloats. That also keeps IntPtr on the "unsafe" side and native-sized numerics on safe side. In this view, native floats would seem to be natural for doing numerical computations (e.g. graphics, geometry, physics etc.). It would even make sense to have nint/nfloat 64-bit on 64-bit CPU when the application itself is 32-bit.

That said, the motivation above seem to be to improve the situation with pointer arithmetics, and in that case I am not entirely convinced introducing a new built-in type is worth saving the keystrokes.

[mihailik]

@jnm2 > And for those reasons, no one does this.

The whole point of "LOB lobby" here is to keep these unsafe ideas out of the language proper:

Likewise, with IntPtr, it is really no different than serializing any other variable sized piece of data (arrays, strings, structs like BigInteger, etc...). You need to store the size of the data and then store the data itself.

[jnm2]

@mihailik Those are not unsafe ideas. @tannergooding is making a reasonable point. It doesn't matter whether you use int, long, byte, nint, BigInteger- use whatever makes the most sense in memory, and when you serialize, serialize in a platform-, endian- and word-size-independent fashion.

[mihailik]

@jnm2 you've missed a tiny insignificant detail man ;-)

You cannot deserialize 64-bit value on 32-bit platform.

[jnm2]

@mihailik Of course not, which is why you use whatever makes the most sense in memory to hold the size of values you are planning for. I'm really not sure what the problem is. If you need values 2^31 or greater, use long or double or decimal or BigInteger or whatever makes the most sense in memory.

[tannergooding]

@miloush, the purpose of extending IntPtr is specifically for platform-sized numerical arithmetic (think size_t). If the user is actually working with pointers/handles/opaque types, then doing actual pointer arithmetic is reasonable (because that is what you are actually working with).

[Drawaes]

Forgetting nint for a second. There is no such thing as a native sized float. Most fpu on cpus are fixed. As said before the fpu on the 8086 was 80bit internally ... having it native sized has one application I have ever heard of and that is the Apple API. I would love to hear of others

[tannergooding]

@mihailik, you also cannot deserialize an array that has 20 elements into a container that can only hold 10 elements.

You are completely correct that you shouldn't be serializing IntPtrs (except in some very niche, low-level, circumstances). You also shouldn't be serializing pointers, the result from object.GetHashCode(), and you should salt your passwords.

This doesn't stop badly designed LOB apps (or code in general) from being written to do as such and neither will it prevent users from doing as such just by throwing it behind some switch.

The purpose of this proposal is to fill a very specific (and very much needed) scenario for Interop code. Having nint in your code is no more unsafe than using ref, out, ref readonly, extern, etc. And regardless of whether any particular platform drops support for 32-bit, the APIs that need to be interoped with (as declared) are still using a variable sized integer, the size of which could change in the future as hardware progresses (not to mention all the legacy codebases that will still exist and need to run/be interoped with).

[mihailik]

@jnm2 serializing and then deserializing a value of platform-specific type inherently leads to data corruption. Please reach me offline (on gmail) for further discussion, to avoid messing up the conversation.

[tannergooding]

@Drawaes, correct.

The only known use today (to my knowledge) is for better interop with Mac/iOS. It is also different from nint in that it has no current runtime support (no underlying runtime primitive, handling, etc).

It does fall into a similar discussion in that it is a needed interop type. However, due to their being no underlying type it comes down to only being able to be implemented via a struct (rather than compiler magic -- unless the runtime were to actually add support). Being implemented as a struct comes with all kinds of downsides (no constants, no checked/unchecked, etc). As such, the conversation about nfloat will likely become even more involved (although possibly not as controversial 😄)

[mihailik]

@tannergooding why don't you share that "very specific (and very much needed) scenario" and we avoid unnecessary hypothetising?

[tannergooding]

@mihailik, I've listed it multiple times. Ease of interop with the underlying platform APIs (especially for Android, iOS, and Mac).

As you've stated:

You cannot deserialize 64-bit value on 32-bit platform.

In the same vein, you cannot deserialize a 128-bit value on a 64-bit platform. Because of this condition, you cannot just create managed APIs (which wrap the platform APIs) that use anything other than IntPtr as it causes a future-compat concern. That is, if we ever gain hardware where the underlying platform size is larger than 64-bit, then all managed code now has to be rewritten/recompiled, rather than just working. We have already seen the difficulties and fallout of this in the transition from 16->32-bit and again for 32->64-bit.

Even under the argument that 64-bit is all we will ever need, that still leaves all of the 32-bit devices where code that wraps IntPtr now has to do twice the work for every simple add/sub operation (and significantly more work for mul/div operations). A lot of these 32-bit devices are embedded or mobile devices where battery life matters.

For better or worse, we've gotten by so far due to the shape of the Windows APIs we have had to wrap so-far. They are designed in such a way that most variable-sized types are actually either a handle or a 32-bit value (and never actually a size_t). They were also written in type-unsafe languages where this kind of stuff is allowed/easy to do.

The other platforms that now need support aren't designed the same. They often take an NSInteger and have it actually represent a platform-sized numeric value. They are often written in type-safe (or at least less unsafe) languages.

Outside of the platform APIs, there are other interop scenarios that fit the same bill (such as multimedia graphics applications, games, etc). The lowest level of code will be unsafe and directly wrap the underlying native code. It will then expose those APIs to higher level code in a safe manner. Even when done in a safe manner, the "safe" APIs still need to expose fields/properties as IntPtr when the underlying type is something like size_t.

Just because I'm writing my own game using framework X, which itself wraps Vulkan, doesn't mean that I should have to use unsafe code to interact with their APIs that expose the size of an array using nint.

I should also be able to easily interop with other code (F#) that already supports all of the appropriate operators on IntPtr.

Native sized numbers is something that no one can get away from because they are a part of the underlying platform (both hardware and software). We can try to abstract them away, but they are something everyone has to deal with and be aware of to some degree.

The only reason you don't already have to work with nint on a regular basis is because the runtime decided that the size of an array is capped at Int32.MaxValue. This itself has caused various issues in places over the years that have had to be worked around (there were issues working with streams larger than 4Gb for a while).

[mihailik]

@tannergooding when we discuss "very specific" scenario in context of compiler design, it's customary to present code samples. Is it something you can do?

[tannergooding]

https://github.com/search?q=org%3Axamarin+nint&type=Code

[tannergooding]

It is also obvious that we are going in circles and neither of us will ever convince the other of our POV. I believe most of the relevant arguments have been laid out (from both sides at this point) at this point and we probably won't get much further without derailing the issue entirely (although we might have done so already :smile:)

[mellinoe]

Forgetting nint for a second. There is no such thing as a native sized float. Most fpu on cpus are fixed. As said before the fpu on the 8086 was 80bit internally ... having it native sized has one application I have ever heard of and that is the Apple API. I would love to hear of others

Without jumping into the other parts of the conversation, I just wanted to say that I hope this part doesn't get lost in the noise. I think NativeFloat is misguided, in name at the very least.

[OtherCrashOverride]

What is the premise that nint is somehow for interop based on? Intptr currently serves this function. nint would be redundant for interop. What we need is nlong aka native long so interop can work on LP64 and LLP64 platforms. All the world is the former. Windows is the latter. This makes it the only item of everything proposed that is actually relevant to modern Mac/iOS. Yet, somehow, its also the only item being completely ignored.

[mihailik]

@tannergooding I see no relevance in your last statement. Language features need to come with justification that's more of a hard science than personal preference. That's why examples are useful.

Thanks for your effort of picking those! The top example you've picked (thanks again!) illustrates the value of this feature in practice:

```c#
using Alias2 = global::System.nfloat;
using Alias3 = nfloat;

namespace MonoTouch.Whatever {

    enum NintEnum : nint { }
    enum NuintEnum : nuint { }

    class Foo {
        nint x;
        nuint y;
    }
}

```

To me it's crystal clear exactly how much value nint introduces here. Hope it helps to settle the matter!

[markusschaber]

@OtherCrashOverride I agree with you here...

Somehow, it seems that this Proposal focuses on int type for interop - however, on all relevant platforms, an int in C is 32 bit nowadays.

We currently use p/invoke to call into a C interface for a C++ library (which cannot change it's type declarations as they need to stay binary compatible with existing clients). They expose masses of long and unsigned long parameters, return values and struct members.

Now, the problem is, that while long is 32 bit on Windows x86 and x64, as well as Linux x86, it's 64 bit wide on linux x64. (We did not yet check ARM, or MacOS etc...)

So as there's no single datatype we can use, so we currently define most structs and p/invoke signatures twice, using if() to guard the calls in higher level wrapper code.

[ericwj]

These full type names don't feel quite 'ECMA 335'-ish imho.

[eerhardt]

Closing as nint and nuint have been implemented in .NET 5.

[dsyme]

@eerhardt Please link to the design docs, implementation etc.

[jaredpar]

@dsyme here is the C# language spec for them https://github.com/dotnet/csharplang/blob/master/proposals/csharp-9.0/native-integers.md. It's effectively just exposing the underlying CLR native int tat's been around since 1.0 in the language much as we expose the other primitive types. Nothing changed about the runtime here.

[dsyme]

I see, thank you. Yes, F# has had this since v0.5. Good to see C# catching up.

Cc @cartermp @TIHan