Runtime: Issues with build -tests and build-tests

[danmosemsft]

I don't have a unix box handy right now, but it looks like it needs updates eg it looks for folders named 'netcoreapp1.1'

I did the .cmd -- https://github.com/dotnet/corefx/pull/15811

[weshaggard]

We should try and get this to use the same logic as build-tests.sh and just skip the building part and go directly to running.

[danmosemsft]

We can delete run-test.sh but there needs to be a way to build without running tests. Is therea flag on build-tests that I missed?

[danmosemsft]

And run without building.

[weshaggard]

There isn't today but I think we can probably figure out a way to make that scenario work.

[danmosemsft]

build -tests apparently doesn't run tests.
build-tests does run tests, but there should be a way to build without tests.
@Petermarcu

[Petermarcu]

I was running build.cmd -BuildTests I didn't try build.cmd -tests

[weshaggard]

I just added `build.cmd -tests" in PR https://github.com/dotnet/corefx/pull/16207 which will build and run tests on top of building the src.

[weshaggard]

We do still need an option for build-tests.cmd that will just do a test run.

[ViktorHofer]

@weshaggard do we still want the option to just run tests without building them?

[weshaggard]

I do think it is probably worth having an option to be able to run tests. As part of the arcade effort we may do away with build-tests.cmd/sh but we will still need some options for building and running tests so lets figure out something for this.

[wtgodbe]

@safern @ViktorHofer what does the story look like for building/running tests look like in CoreFx now?

[safern]

In order to build tests we can do:

1. Build sources + tests (just build, not run tests) with -includetests flag and /p:SkipTests=true . i.e: build.cmd -includetests /p:SkipTests=true
2. Build only tests: build.cmd -test /p:SkipTests=true
3. Build sources and tests, then run tests, like option 1 but without /p:SkipTests=true flag.
4. Build and run tests, option 2 without /p:SkipTests=true

However we don't yet have an option to just run tests without building the whole test tree.

I think we should enable that option, and also consolidate the other options, it feels really weird and odd to me when I read a command with -tests /p:SkipTests=true it feels kind of confusing if you don't know what each flag means.

[ViktorHofer]

Fair assessment. Something that I discovered recently and discussed with @danmosemsft is that arcade offers more options that we currently don't use:

  [switch] $test,
  [switch] $integrationTest,
  [switch] $performanceTest,

@safern do you know out of your head if an arcade enabled repo has an option to just run the tests put aside corefx which is still special? maybe with the $execute switch?

[safern]

I think arcade's -test option from looking at the code, what does is calling the underlying project with /t:test so for not special repos, that will cause the tests to just run, and I think that is the intention of that option.

However in corefx, the underlying project that is called is build.proj which overrides the test target, and calls src\tests.builds which is the one that always build the whole test tree. I think we should fix tests.builds to enable just running tests, integrationTests and performanceTests, using the targets that arcade would call with those options, or by passing extra properties in our build scripts that tests.builds knows and does the right set of things.

[ViktorHofer]

Thanks. I will add this to our Arcade epic as it breaks the promise of what Arcade stands for.