Sdk: [ffi] Add support for abi-specific integer sizes

[mkustermann]

We should consider exposing our Abi enum and allow users to have ABI-specific types. This would avoid us having to continuously increase the set of types we support in the core dart:ffi library. It would also allow a ffi bindings generator to produce ABI agnostic bindings.

We would support the primitive integer, double and pointer types in dart:ffi anything else can be user defined.

One option would be using non-function type aliases (see https://github.com/dart-lang/language/issues/65), which could look like this:

typedef WChar = AbiSpecificType<Int8, Int16, ...>;

class MyStruct extends Struct {
  @WChar()
  int char;
}

void foo(Pointer<WChar> chars);

where dart:ffi would have

class AbiSpecificType<ABI1, ABI2, ...> {}

When the VM compiles code it will know which ABI it runs on and can use the corresponding type from the instantiated AbiSpecificType.

One downside here is that we would have a hard time adding more ABIs in the future (though in reality we rarely add new architectures / operating systems / compilers)

/cc @dcharkes wdyt?

[mkustermann]

@dcharkes There is actually a way to make this more extensible to future ABIs:

abstract class WChar implements LinuxArmHardFp<Int8>,
                                LinuxArmSoftFp<Int16>,
                                WindowsX64FastCallAbi<Int32>,
                                ... {}

Adding a new Abi would then just add a new class to dart:ffi which is not breaking compatibility. When running the program on an Abi that WChar was not implemented the VM could throw an exception (or a compile-time error).

[dcharkes]

I like it! We might still need to make it extend AbiSpecificInteger or make all those individual ABIs do that, in order to be able to have Pointer<WChar> and be able to store/load from it through extension methods.

This also means we would not have to wait on generalized typedefs (or result to subtyping) to be able to reuse the type in multiple places.

[mkustermann]

There's also a design choice: In C code two typedefs are compatible and assignable to each other. If we do the generalized typedef approach we will have the same behavior, if we go with a separate class then one needs to explicitly cast in Dart (but not in C).

[dcharkes]

True, if the one package decides to call wchar_t WChar and another package decides on WCharT they wont match.

Maybe we should preempt that by adding some of those to package:ffi, and have people make PRs for adding more common types.

[dcharkes]

We also need something for structs. We cannot merge the two concepts in one Dart type, because the int loads/stores only work for ABI-sized integers, and the struct field loads/stores only work for structs.

[mkustermann]

Not sure how common it is in C to have a typedef to be integer typed on one platform and non-integer on another. Maybe that is not worth adding support for.

[artob]

abstract class WChar implements LinuxArmHardFp<Int8>,
                                LinuxArmSoftFp<Int16>,
                                WindowsX64FastCallAbi<Int32>,
                                ... {}

Adding a new Abi would then just add a new class to dart:ffi which is not breaking compatibility. When running the program on an Abi that WChar was not implemented the VM could throw an exception (or a compile-time error).

That looks pretty great.

True, if the one package decides to call wchar_t WChar and another package decides on WCharT they wont match.

Maybe we should preempt that by adding some of those to package:ffi, and have people make PRs for adding more common types.

Yes, these really should be provided by the Dart project in one form or another. You'd want to avoid the situation with e.g. Java JNA when there's a bunch of different definitions of the same common types (e.g., ssize_t), and nobody knows which type they ought to be using.

[dcharkes]

@mkustermann The typedef solution does not work for struct fields (annotation constructor calls cannot have type arguments), so that solution also requires subtyping.

I've prototyped three API variants in https://gist.github.com/dcharkes/8d41b7def9bf82c74bf0b7f8a1d66a6c.

@lrhn @eernstg Any feedback from a language design point of view on the possible APIs?

[mraleph]

Just to make sure we are considering this. Original FFI Vision document featured "dictionary" based portability design. I think we should consider going the same way instead of trying to use types to express things which are not types (this makes it somewhat C++esque). More concretely we could use something along the lines of:

enum Os { Linux, Windows, MacOS }
enum Cpu { X64, IA32, ARM32, ARM64 }
enum FP { Soft, Hard }

class IfAbi {
  final Os os;
  final Cpu cpu;
  final FP fp;
  const IfAbi({this.os, this.cpu, this.fp});
}

class Typedef {
  const Typedef(Map<IfAbi, Type> mapping, {Type default_});
}

@Typedef({
  IfAbi(os: Os.Linux, cpu: Cpu.X64): Int64,
  IfAbi(os: Os.Windows, cpu: Cpu.X64): Int32,
}, default_: Int32)
abstract class WChar {}

Similar approach can be taken to ABI specific structure layouts.

[lrhn]

I like @mraleph's approach, but I'm wary about using enums for something which isn't naturally bounded. There are more than three operatings systems, and more than four CPUs. We might not support them yet, but baking the list into our libraries makes it harder to expand for third-party users. If they make the SDK compile on FreeBSD on PowerPC, they would still have to edit all the right places to make it recognizable. (Does X32 count as a CPU?)

So, I think I'd rather use strings. Maybe skip the IfAbi class too, and allow free-form patterns:

@Typedef(Int32, {
   {os: "linux", cpu: "x64"}: Int64,
   {os: "windows", cpu: "x64"}: Int64,
   {os: "bsd", cpu: "power" : Int64
})
...

Then there'd be some way to figure out the current configuration, and perhaps even override it.
Maybe allow -Dffi.os=windows to override the os detection for testing.

(On the other hand, I don't want to introduce a new micro-language for specifying constraints like "osVersion": "<10", which is where this could end if taken too far).

[mkustermann]

Annotations work, yes. Not sure why I wasn't thinking of them.

One downside is that one cannot enforce the constraint that all of the possible native types for the typedef should be _NativeIntegers. In case of WChar we want to ensure it is an integer type, so we can write

class Foo extends ffi.Struct {
  @WChar
  int character;
}

I guess if we wanted to surface this we would need to have the special analyzer + CFE logic to issue additional compile-time errors based on examining the evaluated typedef constant annotation.

There's also the question whether we want to support mixed type definitions (e.g. typedefs where it's a pointer on one OS and integer on OS). The types in Dart would then be dynamic I assume and the program has to test abi at runtime and do a dynamic cast (In C code accessing such fields would be if/def'ed out)

[mraleph]

One downside is that one cannot enforce the constraint that all of the possible native types for the typedef should be _NativeIntegers.

It is true that we would not be able to enforce this requirement with just through normal Dart typesystem.

But in any approach there would need to be some logic _somewhere_ actually evaluating what WChar stands for - that logic can do enforcement of type consistency.

This unfortunately means we would need analyzer changes - which maybe otherwise would not be required.

I also realised that

class Foo extends ffi.Struct {
  @WChar
  int character;
}

would not actually work out of the box in either approaches. You need to do:

@Typedef({...})
class WChar {
  const WChar();
}

@WChar()
int character;

Which is a bit of mouthful - but I think the best we can get now. If only we could use annotations on function type parameters, then we could just go for:

const WChar = Typedef({...});

But this approach sadly does not work with our approach to the NativeFunction<...> type.

@lrhn

So, I think I'd rather use strings. Maybe skip the IfAbi class too, and allow free-form patterns:

My original design was using simple language for conditions - but I agree that that would be too much. The idea behind using IfAbi was to make it self documenting. Using maps as conditions is an interesting variation, though one would have to write:

// Keys need to be strings
{'os': "linux", 'cpu': "x64"}: Int64

// Alternatively we could have markers
enum AbiKey { os, cpu, fp }
{os: "linux", cpu: "x64" } // Map<AbiKey, String>

[dcharkes]

Abi-specific types use

Note that the ABI-specific types also need to extend NativeType in order to use them in function signatures, and that ABI-specific integer types need to extend AbiSpecificInteger in order to be able to use them in loads and stores.

extension AbiSpecificIntegerPointer on Pointer<AbiSpecificInteger> {
  int get value => 0;
  void set value(int value) {}
  int operator [](int index) => 0;
  void operator []=(int index, int value) {}
}

In short, the user-defined types and consts need to work in:

• Pointer type arguments
• native function signatures
• struct field annotations
• loads / stores (for integers)
• field loads / stores (for structs #42816)

Whether the definition per architecture is defined as type arguments, const fields, or annotations does not matter too much for that functionality.

Abi-specific types definition

Using types, const fields, or annotation matters for how much manual checking (with manual error messages) we want to add.

It also matters for whether we have an open-world vs closed-world assumption, and whether we require the user defining the type to define it for all known Dart target ABIs. I see three options here:

• Closed-world, user has to define the type for all target ABIs:
  
  
  
  • We know all types are provided. (Though, the user might fill in garbage for ABIs he does not know about, causing segfaults.)
  
  
  • We need to do a breaking change when we add a target ABI. Or as @lrhn mentioned when someone else targets Dart to a new ABI.
  
  
  • We can enforce this without extra rules in the type system.
  
  
• Closed-world, user can define types for a subset of target ABIs:
  
  
  
  • Trying to run your FFI code on a different target ABI might cause compile errors because of missing typedefs.
  
  
  • We error on target ABIs we don't know about, preventing typos in the definition. (Using enums or types helps here.)
  
  
  • Anyone extending Dart to a new target ABI would need to extend the closed-world.
  
  
• Open-world, user can define types for a subset of target ABIs:
  
  
  
  • Same, but we don't error on unknown ABIs.
  
  
  • This is fragile, a mistype of target ABI that the user is not testing on will go unnoticed.