Chapel: Proposal: Add a Chapel Preprocessor

[LouisJenkinsCS]

I believe that there should be a simple preprocessor, similar to C's preprocessor, that is capable of doing things that a normal compile-time param function cannot. I think that there can really cool interactions between the preprocessor and param functions... It could even support things like recursion and be entirely Turing complete. For example, as mentioned in #9788, the ability to "stringify" an expression, rather than evaluating it, for debugging purposes. By "stringifying an expression", if the user were to write something like assert(ptr != nil), it should be possible to both A) evaluate the expression, which is used to determine whether or not an assertion has failed, and B) a way to turn the expression ptr != nil into the string-equivalent "ptr != nil". I believe that a preprocessor would be wonderful for various other reasons, in which include, but is not limited to...

Example 1) Adding methods to preexisting types

pragma "begin preprocessor"
#define ATOMIC_MAX(x) \
proc x.atomic_max(y) { \
   on this do while true { \
      var _val = read(); \
      if _val >= y || x.compareExchangeWeak(_val, y) then break; \
      chpl_task_yield(); \
   } \
}
pragma "end preprocessor"

ATOMIC_MAX(atomic int(8));
ATOMIC_MAX(atomic int(16));
ATOMIC_MAX(atomic int(32));
ATOMIC_MAX(atomic int(64));
ATOMIC_MAX(atomic real(32));
ATOMIC_MAX(atomic real(64));

In this case, we can avoid having to have literal-copy-pasted functions that all do the same thing, similar to Atomics.chpl. Imagine that this can be done for any type.

Example 2) Acquiring line number and file name for debugging purposes

userDebugFn(msg, CHPL_LINE, CHPL_FILE);

How will this be implemented

As seen above, I'm thinking we can just have a very simplistic preprocessor that does almost the same thing that C's preprocessor does. In this case, it does what param functions cannot by copy-pasting code; it is not meant to replace param functions.

[bradcray]

I'm not wild about this proposal for the following reason: When I think about things that preprocessors do, they tend to fall into one of two cases for me: either (1) we should improve the language to support them or (2) existing preprocessors (e.g., m4) should be used instead of inventing a new one. Specifically, I feel allergic to the notion of creating a Chapel-like language that requires a language-specific preprocessor to feel complete because it makes the language seem incomplete, because it's hard enough to design and implement a new language and compiler without adding a new tool on top of it, and because I think there are good preprocessor tools in existence already.

With respect to the specific examples above:

• while I think ultimately having Chapel support stringification of its own syntax is attractive, I think it's a ways off in the future, so would ask what it would take to do something like this in m4 when needed for now

• I'd hope that adding methods to pre-existing types could be done through the language's generics machinery for many cases such as the one shown here; if not, again I wonder if m4 would be up to it.

• I've long believed that there should be built-in param identifiers or functions in the compiler for referring to the source line number and filename (the compiler knows it, there's no reason it couldn't / shouldn't expose it). I think this is simply low-hanging fruit that nobody's ever gotten to...

[ronawho]

For Example 1) with atomics -- the only reason the copy pasted code existed was because of C limitations. C doesn't support overloads so we had to give each atomic function a unique name by embedding the type name. The runtime defines the atomic functions with macros, but in the chapel code we needed to call out to the unique names and we did this with distinct types. #10298 added support for computing extern proc names, which allowed us to clean up the atomic implementation in #10588

With that you can now just write something like:

// processor atomic max() -- defined for all int/uint/real types
proc AtomicT.max(other: this.T) {
  on this {
    var curMax = this.read();
    while curMax < other && !this.compareExchangeWeak(curMax, other) do
      curMax = this.read();
  }
}

var a: atomic int;
a.max(5); writeln(a);      // 5

var ar: atomic real;
ar.max(4.0); writeln(ar);  // 4.0
ar.max(1.0); writeln(ar);  // 4.0

IMO a C-like preprocessor would be a kludge for any modern language, and I strongly agree with Brad's notion that needing a preprocessor would make Chapel feel incomplete.

I'd be in favor of closing this issue and maybe opening a new issue about implementing support for file/lineno queries from example 2)

[LouisJenkinsCS]

I'm impressed, you really did significantly clean up the Atomics implementation while also providing new functionality, nice job @ronawho! 👍

Its really nice, but can it do this?

proc +(x : AtomicT(?T), y : T) : T where isNumeric(T) {
   return x.peek() + y;
}
proc +=(x : AtomicT(?T), y : T) where isNumeric(T) {
   while true {
      var _x = x.peek();
     if x.compareExchangeWeak(_x, y) then return;
     else chpl_task_yield();
   }
}

Etc. I know this isn't entirely on-topic, but it is somewhat related to the desire of C macros, as you could do something like this with a preprocessor, I just wonder if Chapel can do the same... if so it would be so cool if atomics could be used like non-atomic types! (I'll close issue once I get an answer though, but I can open another issue requesting this feature if the answer is no)

[ronawho]

You already have an issue for supporting operators on atomics -- https://github.com/chapel-lang/chapel/issues/8847. It is also in https://github.com/chapel-lang/chapel/issues/9368 as "operator overloads for atomic ops".

But yes, this change makes it easier to implement operator overloads, though it was possible before (note that Chapel has always supported operators overloads and that operator overloading is unrelated to having a preprocessor.)

Today, operators are explicitly disallowed in Atomics.chpl module, but you can write your own similar to what you had above:

inline proc +=(ref a: AtomicT, b) {
  a.add(b);
}

var a: atomic int;
a += 1;
writeln(a);           // 1

var ar: atomic real;
ar += 4.0;
writeln(ar);          // 4.0

We have plans to implement operator overloads ourselves, but to start we're focusing on any backwards incompatible changes (https://github.com/chapel-lang/chapel/issues/10001), and operator overloads are not backwards incompatible.

[LouisJenkinsCS]

Fair enough. Nice job on the overhaul