Omr: Proposal: New directory structure for OMR Compiler tests

[Leonardo2718]

Currently, OMR compiler testing is somewhat disorganized. The Test Compiler component is located in fvtest/compilertest while its related tests are in fvtest/compilertest/tests. Tril is implemented in fvtest/tril with some tests in fvtest/tril/test and Tril-based compiler tests are in fvtest/compilertriltest. I would like to propose a restructuring of the directories related to compiler testing as follows:

• fvtest/compilertest/: top-level directory containing everything related to testing the OMR compiler
  
  
  
  • testcompiler/: Test Compiler implementation
  
  
  • tril/: Tril implementation
  
  
  • tests/
  
  
  • il: Tril-based IL opcode tests currently in fvtest/compilertriltest
  
  
  • optimization: Tril-based tests for certain optimizations currently in fvtest/compilertriltest
  
  
  • ilinjectors: IlInjector-based Test Compiler tests currently in fvtest/compilertest/tests
  
  
  • tril: Tril implementation tests currently in fvtest/tril/test
  
  
  • more directories for other compiler component tests (e.g. for new RawAllocator from #3480)
  
  

[Leonardo2718]

I would appreciate feedback on this proposal from @0xdaryl, @fjeremic, and @mstoodle.

@knn-k @dchopra001 @aarongraham9 @hsoontie @marufunb may also be interested in this.

[fjeremic]

I find there is a lot of redundancy in the directory names. A few things that have bugged me:

• Why is fvtest not simply named test as this is a lot more intuitive?
  
  
  
  • I suspect most newcomes to the project probably look for a test directory and might not see fvtest
  
  
  • Not sure functional verification has much meaning here
  
  
• Many of the test directories have "test" in the suffix, etc. which seems unnecessary
  
  
  
  • If they are all placed in the root test directory then we can shorten quite a few of the names
  
  
• Do we have an initiative to deprecate the Test Compiler in favor of far superior tril testing?

[mstoodle]

i had a similar kind of reaction to fvtest/compilertest/testcompiler :) . I like the suggestion to rename fvtest to test but worry that this directory name has a high likelihood to be hard-coded into a plethora of scripts. So long as those scripts all reside in our git repo it should be fine to rename, but somehow I'm a bit dubious that will turn out to be the case...

I wonder if we should separate the implementation of different compiler test frameworks (compilertest, tril) from the tests themselves? There are a few choices: 1) promote them to their own top-level directory like jitbuilder, 2) create a new top-level directory to house them, something like testinfra/ maybe (which could also house other test "infrastructure" needed by other components, or 3) create a separate directory like fvtest/compiler/infra and then have the different test directories @Leonardo2718 outlined appearing beside it inside fvtest/compiler . We start getting into pretty deep paths with that third one, though.

Oh, and 4 through N) do something else :) .

[Leonardo2718]

• Why is fvtest not simply named test as this is a lot more intuitive?

  • I suspect most newcomes to the project probably look for a test directory and might not see fvtest
  • Not sure functional verification has much meaning here

• Many of the test directories have "test" in the suffix, etc. which seems unnecessary

  • If they are all placed in the root test directory then we can shorten quite a few of the names

I like the suggestion to rename fvtest to test but worry that this directory name has a high likelihood to be hard-coded into a plethora of scripts. So long as those scripts all reside in our git repo it should be fine to rename, but somehow I'm a bit dubious that will turn out to be the case...

I like that suggestion as well. However, as @mstoodle pointed out, there is likely going to other things that have to be changed as well if we rename fvtest to test. For now, I only want to consolidate compiler-related testing in an effort to make future changes easier.

I had a similar kind of reaction to fvtest/compilertest/testcompiler :)

Heh, yeah I can appreciate that :smile:

• Do we have an initiative to deprecate the Test Compiler in favor of far superior tril testing?

Yes! I'm hoping to start removing Test Compiler tests that already have Tril counterparts soon. There are still some tests that haven't been migrated to Tril so removing themt will take a bit of time. The Test Compiler itself is going to be re-purposed as a new backend for Tril (JitBuilder and Tril are starting to have conflicting goals).

I wonder if we should separate the implementation of different compiler test frameworks (compilertest, tril) from the tests themselves? There are a few choices: 1) promote them to their own top-level directory like jitbuilder, 2) create a new top-level directory to house them, something like testinfra/ maybe (which could also house other test "infrastructure" needed by other components, or 3) create a separate directory like fvtest/compiler/infra and then have the different test directories @Leonardo2718 outlined appearing beside it inside fvtest/compiler. We start getting into pretty deep paths with that third one, though.

Yes, this is something I've thought about. Of the options listed, I prefer the second option best. I agree the third option is going to result in rather long paths, which can get cumbersome. I'm also concerned that "promoting" test frameworks to top-level directories is going to make things noisy by mixing together units from different OMR components. Creating a new top-level directory for testing related infrastructure strikes a nice balance between path length and compartmentalization.

Again, with this proposal, I mostly interested in consolidating compiler related testing infrastructure so that future changes will be easier to organize.

[0xdaryl]

@smlambert FYI

[0xdaryl]

I agree with:

• renaming top-level fvtest directory to test in the long-term. It does not need to be part of this particular reorganization though.
• removing the test Smurph™️-ing from the subdirectory names
• a top-level testinfra directory

In addition to the GH repo, you probably want to check into the CI scripts to see if anything needs to change there as a result of any renamings.

In the interests of moving this forward, I suggest you follow this issue up with a pull request with your proposed changes so that reviewers can further comment on real changes.

[smlambert]

Wonderful, I was very recently thinking about a rename of fvtest to test... and general naming cleanup (removing redundant prefix/suffix of test for any contents within the test directory).

There should not be many/any scripts that are reliant on those names, if so, good to break that dependency anyway.

Thanks for this initiative!