Omr: Tril direction and wishlist

[0xdaryl]

Progress on Tril as a means for injecting precise TR IL into an OMR compiler has not made much progress since its introduction over a year ago. This is unfortunate as it is a valuable and important part of the testing story for the compiler and JitBuilder. It is worth laying out at least a short-term roadmap of features with corresponding issues to describe the direction we want this technology to take and so that developers interested in advancing this have some place to start.

The goal with this issue is to collect inputs from OMR contributors to drive and prioritize these requirements.

Here is an initial, but not exaustive, list of ideas in no particular order. Please feel free to add a comment to this issue for any new ideas or suggestions you have. There are issues people have already created in OMR (marked with the Tril label), but they are not all reflected in this issue yet.

• Replace existing fvtest opcode tests with equivalent Tril variants

  • consistent tests across platforms (easier maintenance)
  • immediate support on new platforms (e.g., ARM, AArch64)

• Allow platform and capability-specific tests (#1843)

  • allow tests to be selectively disabled

• Remove dependency on JitBuilder (#1688)

• Persist more of TR environment in Tril and reconstruct

  • SymbolReferenceTable (#1552), Symbols
  • Node flags (#1947)

• Improve Tril coverage of TR IL opcodes, including but not limited to

  • switches (#2000)
  • exceptions/try-catch
  • regdeps
  • vector (#1954)

• Support a strict mode

  • Tril is flexible enough to support the generation and injection of invalid IL to test things like the IlValidator. However, for those writing "normal" test cases (the majority of uses of Tril) it would be useful to generate compiler warnings or errors about invalid tests

• Error handling (#1949)

  • provide more location-aware error info: line number, column, token (#1938)
  • persist through AST to support AST level warnings with same level of error handling

• Support manual setting of block frequencies to influence optimizations

• Support Tril method to Tril method dispatch

• Use Tril to target specific optimizations (or subsets of optimizations)

• Provide gentle, how-to introductory documentation on creating a Tril test case

• Revisit test case inputs

  • does the const_values template provide enough coverage?
  • improve test coverage mixing parameters and const testing

[0xdaryl]

FYI @Leonardo2718 @smlambert

[0xdaryl]

I plan to have a discussion on this at the Nov. 21 compiler architecture meeting.

[knn-k]

Provide gentle, how-to introductory documentation on creating a Tril test case

I would like this to be provided, as I just started contributing test cases.

[charliegracie]

The first step would be to modify the Tril code so that it can be compiled on ALL of the OMR platforms. I believe some functionality is going to have to be removed or re-written to remove C++ features that are not available on all platforms. Also the current set of tests do not work on 32 bit platforms. The first step for Tril improvements outside of documentation would be to make the Windows 32 bit version work with the current minimum supported compiler used by OMR. This would likely cover most of the basic issues with using Tril on platforms that are not Linux GCC based.

[mgaudet]

A mode in trace logs that emits Tril to allow using Tril to build regression tests for optimizations and code generators would be valuable, though this would involve providing representations of a variety of background data-structures which have no current representation like alias sets.

[mgaudet]

A textual testcase format ala LLVM's lit would also be a valuable addition: http://llvm.org/docs/CommandGuide/lit.html (http://llvm.org/docs/TestingGuide.html provides some information on how it works today, and how it is used)

[mgaudet]

There's also an interesting question about how to write test cases that exercise dynamic features: Like how would you write a test case that exercised runtime assumption invalidation, using tril.

[smlambert]

When the initial opcode tests were created (just around the time JitBuilder came into existence), we asked about replacing existing fvtest opcode tests with equivalent jitbuilder-y approach (speaking to your first bullet point above). At that time, it was mentioned that it was not do-able because jitbuilder created extraneous IL that we did not want to inject (and is avoided when hand-crafting the IL to inject, as the current opcode tests do).

Is that no longer the case (that extra IL is generated and/or we no longer care if it is injected?), because it would most certainly be nicer to write and maintain tests that are created on the 'slightly higher layer' of Jitbuilder/TRIL.

Apologies that I've not paid close attention to the evolution of TRIL, hopefully this question is still relevant, but will be glad to get caught up on the state of the art, if not.

[Leonardo2718]

@smlambert

Is that no longer the case (that extra IL is generated and/or we no longer care if it is injected?), [...]

Although JitBuilder now generats much "cleaner" IL than before, it still doesn't provide the level of control that's needed for testing purposes (and that's intentional, of course).

Tril uses JitBuilder to compile the IL, but not to generate it. It instead manipulates the IL directly and uses IlInjector to "piece it together". So, it allows precise control over what IL is generated while also being clean and easy(er) to maintain.

I actually want to replace JitBuilder with something that provides better control over the compilation process so we can also precisely specify what part(s) of the compiler is to be exercised (see point 3). Doing so shouldn't affect how the existing Tril-based tests work but would allow us to write better/more interesting tests later on.

Does this answer your question? :slightly_smiling_face:

[...] because it would most certainly be nicer to write and maintain tests that are created on the 'slightly higher layer' of Jitbuilder/TRIL.

100% with you on this one :smile:

[smlambert]

Answers my question wonderfully, thanks @Leonardo2718 !

[fjeremic]

From the currently proposed list the following are in my opinion most important to uses of Tril I've seen, sorted by order of impotence (most important at the top, least important at the bottom). Some items did not make the cut.

• Replace existing fvtest opcode tests with equivalent Tril variants
• Improve Tril coverage of TR IL opcodes
• Provide gentle, how-to introductory documentation on creating a Tril test case
• Allow platform and capability-specific tests
• Support manual setting of block frequencies to influence optimizations
• Persist more of TR environment in Tril and reconstruct
• Use Tril to target specific optimizations (or subsets of optimizations)
• Remove dependency on JitBuilder