Chapel: tuples not a supported type in exported functions

[kat-young13]

Ran into an error when trying to declare a tuple in an exported function!

export proc createGeneticOptimizer(evaluator_idx: int,
                                     generations: int = 1000,
                                     gens_per_epoch: int = 1,
                                     num_demes: int = 4,
                                     pop_size: int = 64,
                                     mutation_rate: real = 0.05,
                     mutation_degree_mul_small: 2*real = (0.0, 0.2),
                                     crossover_rate: real = 0.33,
                                     migration_interval: int = 5,
                                     in log_fn: c_string = c"",
                                     verbose: bool = false)

[ben-albrecht]

@kat-young13 - do you have a minimal example demonstrating this? Is this for argument types and/or return types?

[bradcray]

Also, can you say what language you were hoping to export the function to, and how you wanted the tuple to appear in the other language? (e.g., what type did you want it to appear as in Python/C/whatever?)

And: would having this feature unblock something you're working on, or do you have/need a workaround?

[ben-albrecht]

And: would having this feature unblock something you're working on, or do you have/need a workaround?

Assuming this stems from https://github.com/Cray/crayai/issues/209 (@kat-young13 - correct me if I'm wrong), I am confident we can work around it using other supported collection types, i.e. arrays.

[lydia-duncan]

From the example @kat-young13 showed me, it was an argument and could be gotten around by using just two separate arguments instead of the tuple pair - I think it would be useful to support for her, but it doesn't seem like a blocker

[mppf]

Here is an example

extern {
  #include <inttypes.h>

  void acceptsTuple(int64_t tup[2]);

  void cfunc(void);

  void cfunc(void) {
    int64_t tup[2] = {0,1};
    acceptsTuple(tup);
  }
}

export proc acceptsTuple(in arg:2*int) {
  writeln(arg);
}

This doesn't compile with the C backend with in or const ref intent on the export proc due to some type mismatch errors from the C compiler. However it does compile and run correctly with --llvm.

It is the same situation with c_array:

use CPtr;

extern {
  #include <inttypes.h>

  void acceptsArr(int64_t tup[2]);

  void cfunc(void);

  void cfunc(void) {
    int64_t tup[2] = {0,1};
    acceptsArr(tup);
  }
}

cfunc(); // calls acceptsArr

export proc acceptsArr(const ref arg:c_array(int, 2)) {
  writeln(arg);
}

To fix, I think we'd need to adjust the compiler to use the appropriate C utterances in the type declaration for the export proc we generate instead of something more typical; i.e. instead of

void acceptsArr(c_array_int64_t_2_chpl * arg)

we should generate the C signature

void acceptsArr(int64_t tup[2]);

Note that these are the same in practice because C always passes array arguments as a pointer to the 0th element.

[kat-young13]

Also, can you say what language you were hoping to export the function to, and how you wanted the tuple to appear in the other language? (e.g., what type did you want it to appear as in Python/C/whatever?)

And: would having this feature unblock something you're working on, or do you have/need a workaround?

Python float, and I can work around for sure!

[mppf]

I started to look at addressing the problems with export proc with tuple or c_array arguments but I ran into some challenges. There are two challenges:

1. When it comes down to it, I don't know if the expected C declaration for a 2-tuple of int or 2-long c_array of int is void f(int64_t tup[2]); or void f(int64_t* tup). As far as I know the C programmer might need to know which one it is in case they need to make their own prototype (which I was doing above) - and as far as I know the C compiler will consider them incompatible.
2. The Chapel compiler handles star-tuple and c-array types in special ways because of the way that, in C, a fixed size array declaration is also a pointer to the 0th element.

[bradcray]

in case they need to make their own prototype

I'd expect Chapel to make this call and to announce it through its generated .h file and documentation. In what cases would a C programmer feel compelled to create their own prototype? (since that worries me).

[mppf]

@bradcray - yes, that makes sense in the context of library mode; but there are other use cases for C interop at least (such as the Chapel -> C -> Chapel examples above in https://github.com/chapel-lang/chapel/issues/15944#issuecomment-649800667 ). In some situations C programmers do make their own prototypes for library functions; that usually is something omitted (intentionally or not) from the library header. I don't know of any good reason why that would apply to the header we generate for export procs (since either it's marked export and we generate an entry in the header - or it's not marked export and it isn't callable at all from C).

[bradcray]

I'm not crazy about expecting users to prototype exported Chapel functions in a Chapel -> C -> Chapel call chain either (not that we can stop them, obviously), but didn't realize that we only generate the header file for exported symbols in the event that a --library flag is thrown. That said, I also understand that using extern blocks requires the C code to be parsed in the front-end, not during the "link"-ish step as with other extern features (which might just be another reason for me to not be wild about extern blocks).

Basically, I'd hope to write your motivating example as something more like:

testit.chpl:

require "testit-c.h";

export proc acceptsTuple(in arg:2*int) {
  writeln(arg);
}

cfunc();

testit-c.h:

#ifndef TESTIT_C
#define TESTIT_C

#include <inttypes.h>
#include "lib/testit.h"

static void cfunc(void) {
  int64_t tup[2] = {0,1};
  acceptsTuple(tup);
}

#endif

But back to your original question:

When it comes down to it, I don't know if the expected C declaration for a 2-tuple of int or 2-long c_array of int is void f(int64_t tup[2]); or void f(int64_t* tup). As far as I know the C programmer might need to know which one it is in case they need to make their own prototype (which I was doing above) - and as far as I know the C compiler will consider them incompatible.

My intuition would be to generate it as f(int64_t tup[2]); but uncertainty is why I asked:

can you say ... how you wanted the tuple to appear in the other language? (e.g., what type did you want it to appear as in Python/C/whatever?)

[mppf]

I'm not crazy about expecting users to prototype exported Chapel functions in a Chapel -> C -> Chapel call chain either (not that we can stop them, obviously), but didn't realize that we only generate the header file for exported symbols in the event that a --library flag is thrown. That said, I also understand that using extern blocks requires the C code to be parsed in the front-end, not during the "link"-ish step as with other extern features (which might just be another reason for me to not be wild about extern blocks).

Yeah, I think it would be interesting to consider generating a header file for exports and having it available to extern blocks. There are some compilation-flow reasons that is not easy, but we could open up an issue to talk about it. (One idea would be to instruct the C parser to ignore the bodies of the functions during extern block parsing; at least in the example, then the fact that acceptsArr isn't prototyped yet at that point in compilation might be OK).