Teammates: Show full exception format when running tests from CLI

[LiHaoTan]

The exception format currently shown when running tests using the command line is short (i.e. truncated). This makes tracing problems difficult, especially with UI tests that are not reproducible locally.

Let's change the exception format to full.

[wkurniawan07]

@LiHaoTan just to clarify, you're referring to the single-run tests (test, localTests, failedTests) right?

[LiHaoTan]

@wkurniawan07 I am also thinking that ciTests should be included as well.

Is there an issue I am unaware of? The caveat I know is that too long logs will get cut off by Travis which makes it troublesome (since we may have to download raw logs) but we don't really have that issue right?

[wkurniawan07]

@LiHaoTan There is. Unless it's the final run (in which the exception format is already full), having the full exception has close to zero value.

[LiHaoTan]

@wkurniawan07 In my opinion, I don't think that is the case for some of the tests, especially the UI tests. A lot of them fail randomly halfway and from different causes, and repeating them ends up hiding the different failures.

UI tests can be flaky, end to end even more so but only knowing the reason for the final failure may mask failures from other causes. Correct me if I'm wrong here.

[wkurniawan07]

You're not wrong, but you should not let such be a distraction. Even if they fail at some point for multitude of reasons, if they pass after some try, they are not really worthy of our attention. If you have the time and effort to track such minute details for not really a significant gain, you'll find it better to dedicate that same time and effort to more important stuffs.

And let's not forget the main concern: the logs will easily get blown out of proportion if you need to print all the test failures especially from the HTML tests multiple times.

[LiHaoTan]

I agree with you that as long as they pass after some tries reliably that it is fine.

However, from what I observe they may provide useful information (at least for me) to find out why they are failing after multiple tries, take a modified adapted example:

1. Try 1: Fail because of concurrent database writes
2. Try 2: Fail because browser not in focus
3. Try 3: Fail because button not clickable
4. Try 4: Fail because text box not filled.

And let's not forget the main concern: the logs will easily get blown out of proportion if you need to print all the test failures especially from the HTML tests multiple times.

I am aware of this caveat though while it should not be happening that probably isn't the case all the time but I think that is quite rare for TEAMMATES right?

Sidetrack: Perhaps eventually someday when we can move over to more advanced snapshot testing and have one less thing to maintain (GodMode).

So at least I think the useful information they provide outweighs the caveat, or do you think it's the other way round? If that is the case, then I guess not including the full exception format for ciTests is the right thing to do.

[wkurniawan07]

may provide useful information to find out why they are failing after multiple tries

How often are you going to encounter such case? If you are uncomfortable because "this is theoretically possible and I am itching to fix it", then you yourself should hunt down the case and fix it. You're more than free to set up your own Travis instance with full exception format every time.

For something that happens perhaps only in 1 out of 100 builds, that "useful information" becomes a noise in the other 99 builds.

I think that is quite rare for TEAMMATES right?

You do realize how many people (mis-)use our Travis CI build as a shortcut to catch failures, right?

Sidetrack: Perhaps eventually someday when we move over to more advanced snapshot testing compared to GodMode.

Only then will we be in the position to consider. Don't jump ahead.

[LiHaoTan]

I think that is still very useful information and not noise. From what I see, I think it currently happens like every single build, causing AppVeyor failures, blocking Selenium 3 upgrades but all I have is my intuition to back it up so since you are against this for good reason shall we just go with test, localTests and failedTests?

[wkurniawan07]

happens like every single build

Back to my first reason: does it affect us if some tests goes slightly wrong if they pass eventually? If you're itching to fix them, do it, but perhaps you'll be better off doing something else more productive.

causing AppVeyor failures

??? AppVeyor failures have nothing to do with this.

blocking Selenium 3 upgrades

No one stops anyone from modifying .travis.yml in that PR (or any PR) alone to print the full stack trace early. It becomes somewhat a problem only when it is part of the regular .travis.yml.

shall we just go with test, localTests and failedTests?

That's the right thing to do. Anyway, this is too trivial for a beginner-level issue. You can patch this yourself.

[LiHaoTan]

Ok noted thanks for your feedback.

[LiHaoTan]

@wkurniawan07 I thought about it a bit and I guess there is not a lot of value in adding this.

I presume Gradle truncated the exception format to keep it compact since Gradle is used on the CLI. If someone needs to debug on the command line, then they can turn it on by themselves.