Tvm: [RFC] Namespace for Optimization Specific Annotation

[tqchen]

As we start to get some optimization specific operators to annotate additional information needed by relay. Here is a rough list that I am aware of:

• on_device, annotate which device we should run computation on
• stop_gradient, stop gradient in AD
• stop_fusion, stop to fuse continuous ops, useful for accelerator ware partition
• simulated_quantize, used for intermediate annotation for quantization.

These are not conventional operators like numpy-style or normal nn operator, but are helpful to store additional information in the IR. It would be great to categorize them into the same namespace and provide a unified guideline in terms of support-levels.

Here are some proposals, please feel free to make additional names:

• relay.opt.on_device (opt stands for optimization)
• relay.annotation.on_device (annotation)
• relay.ann.on_device (ann for annotation, alternatively "announcement" :)

As usual, API design is one of the most crucial parts of the development and I would like to invite everyone in the community to participate in this RFC. Let us aim to reach a consensus in a week.

Guideline choices for support level:

• 9, allocate a specific level for annotation
• 10, currently used for experimental

cc @dmlc/tvm-comitter

[zhiics]

@tqchen Thanks for the RFC. I personally would prefer relay.annotation.on_device since some of the passes are probably not for optimization but really for annotation purpose.

As for support level, I would vote for using the separate level 9 for annotation.

[ZihengJiang]

Prefer relay.annotation.on_device

[jroesch]

For these kind of ops I think annotation makes the most sense, we can always choose to further categorize them in the future as we gain more of them. It seems like simulated_quantize belongs else where though? is it really an annotation in the same way that the device ops are?

I do like opt as well, but maybe that is the namespace optimizations should live in.

[zhiics]

@tqchen @jroesch @ZihengJiang
So I think we will go for opening a new annotation namespace.

As for the support level, what do you guys think?

[ZihengJiang]

vote for 10, since it is experimental so we can change it to 9 after it is stable

[tqchen]

The current proposal seems to converge toward annotation, let me suggest that we start with level 10 and move to level 9 once we think things are stable.

Let us leave this RFC open for a week so others have time to take a look and share their thoughts, in the meanwhile, we can proceed with changes in the PR accordingly.

[zhiics]

No problem. I will then go ahead with annotation but keep level 10 for now.

[tqchen]

Thanks, everyone for the helpful discussions. The current consensus is:

• Use annotation as namespace
• Start with level=10, then move toward level 9

@ZihengJiang please also update your PR of quantization accordingly(annotate.simulated_quantize)