Tvm: [RFC] Relay IR Text Format

[joshpoll]

Keyword Proposal

The final keywords are:

• def for global function definitions
• fn for local function definitions/anonymous functions
• if and else for if-else expressions
• let for let bindings
• True and False for booleans
• data types are tvm-style. e.g. float32x4

Loose Ends

There are some elements of the text format that didn't make it into this PR. See #1935 for details.

---

Please comment on syntax design choices for #1781 here!

Notable Syntax Examples

Functions

Named, Typed

def @foo(%x: int64, %y: int64) -> int64 {
    %x + %y
}

Named, Inferred

def @foo(%x, %y) {
    %x + %y
}

Anonymous, Typed

fn (%x: int64, %y: int64) -> int64 {
    %x + %y
}

Anonymous, Inferred

fn (%x, %y) {
    %x + %y
}

Let Expressions and Mutation

Immutable

let %x = 2;
let %x = 3;
let %y = 4;
%x + %y

Side Effects Only

The following are equivalent

print("hello");
1

let %_ = print("hello");
1

Tuples

()
(0,)
(0, 1)

Types

Base Types

Designed to look like NumPy types.

uint32
float32
bool
int32x4
...

Shape Types

unimplemented

(n + 1, 2 * n, 4)

Tensor Annotations

generics unimplemented

def @add(t1: Tensor[s, bt], t2: Tensor[s, bt]) -> Tensor[s,  bt] {
    ...
}

Function Types

fn (int32, int32) -> int32

[tqchen]

Can we move most of the discussion to this issue instead? and post RFC here

[tqchen]

cc @dmlc/tvm-team

[tqchen]

cross posting.
We will need a spec on how nested expressions can be printed,

specifically, instead of

add(%x, add(%y, %z))

We prefer

%1 = add(%y, %z)
%2 = add(%x, %1)

Note that add will become neural network layers such as conv2d etc. When we construct the exprs, it is quite typical that the expression is already CSEed. Example:

%1 = add(%y, %z)
%2 = add(%1, %1)

[MarisaKirisame]

I think we can view it by seeing two kind of let: %1 is the 'explicit let', which map to relay let.
_1 is the 'implicit let', which does not map to relay concept, only to denote whenever _1 is refered, the rhs is used (and shared).

Also to be frank, I do not see a reason to has relay program as graph - we still need let because we have scope and effect(no reference for now, but there is general recursion), we cant see graph as tree (because exponential blowup), and we still need to handle binding as we has let, so we are getting the benefit of neither world.

I had made some facility to automatically turn binder into value (LFunctor), so with it we can operate a tree as graph without actually having one, so I think we should at least sit down and revisit why we want graph right now (we can always add it later).

[joshpoll]

I don't think this concern is within the scope of this issue/PR. IMO the pretty printer should print the AST exactly as it's presented. I think it's too restrictive to force function arguments to be values at parse time, and thus we must allow for nested expressions. We could possibly have a compiler option that forces certain compiler passes like ANF and CSE before printing or even use the pretty printer to reformat code into "graph" style.

[tqchen]

This is an issue because this is how user usually constructs a program via DSL, and we need to make sure when we print, they are print out in a well-defined compact fashion which is readable by the parser(without running any transformation pass). The graph itself is not a graph for the exactly the same reason that it is already CSed as user constructs them.

i.e. if a user constructs the program via DSL(i.e. IRBuilder), she might write

x = some relay_expr
y = relay.op.add(x, 1)
let z = relay.op.add(y, y)

If we simply print out the expression as they are, it will explode as a tree

let %z = add(add(%x, 1), add(%x, 1))

The information that the intermediate value is CSEd is missing here, instead, a better way would be

%1 = add(%x, 1)
let %z = add(%1, %1)

I understand this is normally not considered as good pratice for normal programs, but this makes sense for deep learning workloads, where each operations can be arbitrarily nested.

Most users still think in terms of computational graphs, and it is important for us to support that concept well.

[joshpoll]

Here's a proposal for block syntax per @yuruofeifei's suggestion. It will have the same semantics as ReasonML's for local scope, which is

{
  let %x = 1;
  %x
};
...

desugars to

let _ =
  let %x = 1;
  %x;
...

[tqchen]

cc @junrushao1994 @dmlc/tvm-team @jroesch @MarisaKirisame @yuruofeifei @grwlf please comment on the choices of keywords in the text format. (see beginning of the post)

[tqchen]

I think one general question we want to ask when deciding keyword is consistency, and whether we want to enforce some of these. Note that being consistent with common choices will always reduce the mind burden of users, and we should try to do so when possible

• Consistency with python style
  
  
  
  • boolean true/false vs True/False
  
  
  • tuple: (1) vs (1,)
  
  
• Whether we support alternative printing of list
  
  
  
  • e.g. axis=[1,2] or should we always print Array as axis=(1,2)
  
  
• Consistency with numpy base type convention(and the printing output of tvm types)
  
  
  
  • e,g. float32, bool vs Float32, Bool
  
  
  • I can understand one possible reason to go for Float32 is that it really means Tensor[(), float32], but do we need to make such distinguishment or not
  
  
• Consistent with common choices of existing languages
  
  
  
  • e.g. use of -> vs =>
    
    
    
    
    
    • Rust: fn(args) -> return_type
    
    
    
    • Swift func(args)-> return_type
    
    
    
  
  
• How to express common constant with type information
  
  
  
  • example proposal:
    
    
    
    
    
    • 1 -> const(1, int32)
    
    
    
    • 1.0f -> const(1, float32)
    
    
    
    • float64(1) -> const(1, float64)
    
    
    
  
  

[junrushao1994]

Neat.

BTW, it is necessary to use Int, UInt, Float as we already have Int32, Float32 etc? Will it cause some confusion?

[nhynes]

How to express common constant with type information
Rust has

let float1 = 1f32;
let double = 1.0;
let also_double = 1f64;
// same for {u32, u64, u128, ...}

necessary to use Int, UInt, Float

It will definitely be confusing unless there's a well defined spec for how the compiler chooses to use these annotations. I can see some value of this when compiling one model to different architectures with varied word sizes (e.g., native vs wasm).

tuple

This is already a major point of confusion for novice python devs. Following the convention of tuple = comma will at least keep the confusion consistent.

print Array as axis=(1,2)

Semantically, it's the same thing since the type is fixed at this point. If anything, square brackets _might_ improve readability (e.g., x(y(z, axis=(1, 2)), axis=(3,)) vs x(y(z, axis=[1, 2]), axis=[3]))

[joshpoll]

@junrushao1994 Sorry I should have made this more clear, but Int, UInt, and Float are only used for type constructors, e.g. Float[32, 4] for 32 bit float with 4 lanes.

[junrushao1994]

@joshpoll I see. Sorry I didn't catch that. That looks cool now.

So why not we make type constructors capitalized like Int, Tensor, and normal types uncased like int64, float32?

[joshpoll]

The idea was to have all terms (e.g. variables, functions, literals) use lowercase (this is currently not enforced) and all types use uppercase (this is currently enforced by the parser). This is similar to Python type annotations and makes it clear whether something is a term or a type.

This may become especially important when we have generic shapes like (n, 1, m), because it should hopefully discourage users from writing things like (float32, 1, m).

Moreover, in Python type annotations are lowercase for builtins like int and str, but (by convention) uppercase for classes. I think this is a meaningless distinction for Relay and leads to more problems than it solves. It also can falsely equate functions and types. E.g. str in python is both a type and a function that returns a str.

[joshpoll]

@tqchen @nhynes Since we don't support tensor or list literals, it might make sense to allow both () and [] in many places. On a related note, currently type constructors use [] instead of () like function calls. This was to be consistent with Python conventions and also because things like Tensor[(1, 1, 1), float32] are easier to read than Tensor((1, 1, 1), float32).

[joshpoll]

The use of (1,) instead of (1) is mostly a technical one that could be relaxed in the future. The problem is that (1) can be parsed both as 1 with parentheses and as a tuple with a single element.

[joshpoll]

For functions:

I think using fn for function types is a good idea and an easy change to make.

The reason for using -> over => is both for consistency with Python annotations and also because lines like let %_: () -> Int64 = fn () -> Int64 { 0 }; () are easier to read than let %_: () => Int64 = fn () => Int64 { 0 }; () since => visually interferes with =.

[joshpoll]

I support @nhynes's constant literal annotation suggestion.

[tqchen]

I still think that printing things as Float32 will confuses users. Notably, in many cases, we support type as attributes of functions, for example:

%2 = cast(%1, dtype="float32")

In the above code, the dtype field is printed as a dtype string which is consistent with what user inputs from python, and numpy's type scheme.

While I understand the need for making types consistent, I do feel that if we could align it with their string representation from python side it will reduce users' mental burden. i.e. to support

• Tensor[(1,), float32]
• Tensor[(10, 2), float32x2]
  
  
  
  • Directly use typex to annotate yhe number of lanes, removeFloat[n, m] constructor
  
  

[jroesch]

@tqchen it feels like we should be able to avoid embedding type information in the attributes and instead use the type system.

[tqchen]

@jroesch I agree that in the longer term maybe explicitly annotate the output type is a good alternative. But still, I think it is important to have numpy and python consistency.

Notably, we will have several ways of constructing and print things.

• Construct from python API,
  
  
  
  • e.g. relay.const(numpy.array([1, 2], dtype="float32")) is a valid code
  
  
• Type attributes of numpy style function, relay.zeros((1, 2), dtype="int8")

Users might expect to see similar code being printed out as what they annotated in the arguments.

Similarly for True/False, while it may not be the best choice, but choosing the text to be consistent with python might make it feel cleaner as the python DSL construction code will look exactly like what could be printed out as text.

While I agree with @joshpoll that if we design things from scratch we might choose to just be consistent(all upper case). Having all the base types in lower-case is also not a too bad design choice. As for the case of shape generic variable, since we are already using % prefix for all the variables, likely we will also have similar prefix so these variables won't be confused with the base types anyway.

In short, I think we should favor the principle "be python and numpy consistent as much as possible". I have tried to go against this principle before when designing previous projects, and I had to revert the decision because of confusions caused to the users.

[tqchen]

To follow up with consistency issues, the constant proposal also needs some thoughts.

1.0f32 makes a lot of sense for rust because rust already uses f32 for float32. It may make less sense given the way we choose name types. An alternative could be float32(1.0)

Notably, we only need to print out common types in a friendly text format, that means we likely only have to deal with int32, float/double, bool and rest can be dealt with explicit cast

[MarisaKirisame]

This idea just come out of my head:
can we pin a version number to every relay source/serialized code? It help to switch from one version to another version.

[joshpoll]

@tqchen I think type abbreviations like f32 make sense in Relay as well. C, C++, Java, and Python all have type suffixes for numeric literals. We could also consider forcing all literals to be type-annotated. If we decide to do that I think forcing people to write suffixes makes more sense than forcing people to write wrapping functions.

Additionally I think suffixes are better from a readability perspective.

[joshpoll]

I think using Python conventions for booleans and dtypes is probably the way to go.

[tqchen]

@joshpoll how hard it is to directly implement the parser to support float32x10 and floatx for parameterized types? This way we could remove the base type constructors and keep everything consistent

[joshpoll]

I changed types and bools to be more pythonic.

[joshpoll]

I'm closing this issue since the text format PR has been merged. #1935 contains the missing pieces that didn't make it into #1781 and further discussion about the text format should be moved there.