Sdk: dart:ffi int, long, etc

[dcharkes]

Currently dart:ffi only supports:

• int8_t
• int16_t
• int32_t
• int64_t
• uint8_t
• uint16_t
• uint32_t
• uint64_t
• intptr_t
• float
• double

However, many C APIs use C types such as int, long, and size_t.

We could support these types. The question is if we support these types, then what other types should we support.

An alternative could be to not support these types but provide a way for users to specify types which differ in size per platform.

@jonasfj @sjindel-google

[dcharkes]

sizeof(int) on iOS arm64 returns 4. Thus, we cannot safely substitute int with intptr_t.

[artob]

From a user perspective, the lack of these standard variable-sized types is probably the single largest remaining deficiency in Dart's FFI support. It does rather complicate making FFI bindings for Dart.

As someone routinely working on a number of FFI bindings for a multitude of different languages (most recently, OpenXR bindings), the specific list of missing, absolutely essential variable-sized types in Dart is:

• int and unsigned int
• long and unsigned long
• size_t (stddef.h)

And some other, somewhat rarer variable-sized types to consider supporting would be:

• off_t (sys/types.h)
• ssize_t (sys/types.h)
• wchar_t (stddef.h)

All the aforementioned types listed in priority order, in terms of how badly and frequently I have needed them.

What's the plan for supporting at least int, long, and size_t?

[dcharkes]

These types are not part of the ABI, and the C standard has a very flexible specification (see this quick explanation). So, a compiler can choose what size/alignment these types have.

dart:ffi needs to be able to know what the size/alignment of types is of the compiled shared object loaded into Dart. The DartVM knows what OS (Linux,Windows,MacOS,iOS,Android) and hardware (x64,ia32,arm32,arm64) it is running on, so it can rely on the ABI and use the types specified in the ABI.

However, dart:ffi does not know with what compiler (and what compiler options) the shared object that is being loaded is compiled. So it can not know how to treat these C types. (Adding a possibly endless list of compilers/compiler options to DartVM is probably not a good approach, because it would require us to change the DartVM every time someone tries to use a new combination.)

It might be the case that for some of these types, all the compilers targeting the various ABIs (OS/Hardware combinations) agree on a size/alignment. Then we could incorporate these types into the ABI-logic. The downside is that if ever some new compiler comes up that does not respect the status quo, stuff breaks.

Does anyone have experience with whether the most common compilers agree on these types are handled in specific ABIs? Do we have any prior art in FFIs in other languages?

Another direction we could go to is to pass in the specs for these types into a bindings generator. The downside here is that one needs to re-run the bindings generator for every compiler/compiler options/hardware/OS combination when releasing an app.

@artob, what are you currently doing to work around this limitation?

[artob]

dart:ffi needs to be able to know what the size/alignment of types is of the compiled shared object loaded into Dart. The DartVM knows what OS (Linux,Windows,MacOS,iOS,Android) and hardware (x64,ia32,arm32,arm64) it is running on, so it can rely on the ABI and use the types specified in the ABI.

However, dart:ffi does not know with what compiler (and what compiler options) the shared object that is being loaded is compiled. So it can not know how to treat these C types. (Adding a possibly endless list of compilers/compiler options to DartVM is probably not a good approach, because it would require us to change the DartVM every time someone tries to use a new combination.)

@dcharkes Respectfully, you might be overthinking this. Just about any FFI for any programming language has the same theoretical problem, yet they all provide these FFI types.

It might be the case that for some of these types, all the compilers targeting the various ABIs (OS/Hardware combinations) agree on a size/alignment. Then we could incorporate these types into the ABI-logic. The downside is that if ever some new compiler comes up that does not respect the status quo, stuff breaks.

Does anyone have experience with whether the most common compilers agree on these types are handled in specific ABIs? Do we have any prior art in FFIs in other languages?

Every single FFI in every single programming language I've worked with--with the sole exception of Dart--supports these basic types: at the very least, int, long, and size_t are universal. Off the top of my head, here's a partial list of FFIs I have prior experience with, and how they map the types I mentioned:

• Common Lisp CFFI: :int, :long, :size-t
• Haskell FFI: CInt, CLong, CSize, CSsize, CWchar
• Java JNA: int, NativeLong, Structure.FFIType.size_t, platform.linux.XAttr.ssize_t, char
• Julia: Cint, Clong, Csize_t, Cssize_t, Cwchar_t
• Node.js FFI: int, long, size_t
• Python ctypes: c_int, c_long, c_size_t, c_ssize_t, c_wchar
• OCaml ctypes: int, long, size_t, ssize_t
• Racket FFI: _int, _long, _size, _ssize, _wchar
• Ruby FFI: :int, :long, :size_t, :ssize_t, :wchar_t
• Rust FFI: c_int, c_long, size_t, ssize_t

I omitted Go, LuaJIT, Nim, PHP, and Zig from the list, since their FFI implementations all parse C header syntax directly (and implicitly support these types).

@artob, what are you currently doing to work around this limitation?

I'm not. This, plus #35763, is blocking my efforts for nontrivial FFI bindings for Dart and Flutter. It would be a royal pain in the arse to work around both of these blockers, so I'm focusing my efforts elsewhere for now and hoping to come back to all this after a future Dart release announcement.

[artob]

@artob, what are you currently doing to work around this limitation?

Oh, and I might as well mention this here as an aside given that I haven't found an existing issue for it: the lack of general type aliases--for example, defining CLong as a typedef for either Int32 or Int64 at compile time--also hampers creating FFI bindings.

[dcharkes]

Oh, and I might as well mention this here as an aside given that I haven't found an existing issue for it: the lack of general type aliases--for example, defining CLong as a typedef for either Int32 or Int64 at compile time--also hampers creating FFI bindings.

See https://github.com/dart-lang/language/issues/65 for that.

[dcharkes]

Assuming that the sizes are stable per ABI (so checking a single compiler gcc/clang/msvc per platform) and only for the subset of OS x hardware platforms that DartVM runs on:

Types we already support:

• sizeof(int) == 4 -> use Int32.
• sizeof(unsigned int) == 4 -> use Uint32.
• sizeof(ssize_t) == sizeof(intptr_t) -> use IntPtr.

Types that require dart:ffi support:

• sizeof(size_t) == sizeof(intptr_t) -> Introduce UintPtr (range in Dart 0 - +2^32 on 32 bit, range -2^63 - +2^63 on 64 bit, because Dart does not have unsigned 64 bit ints).
• sizeof(long) == sizeof(intptr_t) except for Windows where sizeof(long) == 4. -> Introduce Long.
• sizeof(unsigned long) == sizeof(intptr_t) except for Windows where sizeof(long) == 4. -> Introduce UnsignedLong (when 8 bytes, will be signed in Dart for the same reason).
• sizeof(off_t) == sizeof(long) -> Introduce Long.
• sizeof(wchar_t) == 4 except for Windows where sizeof(wchar_t) == 2. -> Introduce WChar.

Sources:

• https://godbolt.org/ (linux/msvc and cross compilation to arm/arm64).
• MS documentation about long.
• Manual compilation on macos gist.

TODO:

• Check iOS behavior.
• Check Android arm/arm64 with Linux arm/arm64 behavior.

[artob]

@dcharkes Would you be willing to give any estimate which future Dart release those new types (Long, WChar, etc) might be expected to land in?

[dcharkes]

No, we generally don't give estimates. We're working on it, but we don't want to rush a suboptimal solution (idea for a general solution to ABI-specific types: https://github.com/dart-lang/sdk/issues/42563).