Habitica: Fix super long error output when test fails in Travis build

[khipkin]

I'm not sure why, but lately my builds have been throwing the same error over and over again, sometimes running for more than 45 minutes.

For example, this build only stopped because the output file exceeded the limit of 4MB, with over 80k lines of output. And this only happened after more than 48 minutes.

It's just this failure over and over again:

'build:dev' errored after 28 s
Error in plugin 'run-sequence(test:karma)'
Message:
Command failed: /bin/sh -c NODE_ENV=test npm run test:karma

I think we should limit the number of retries for running karma tests to prevent this from happening again and hogging TravisCI resources.

[crookedneighbor]

I don't think this is a problem with Karma. Pretty sure that the same crazy output will happen when _any_ of the test suites fail.

Changing title to "Fix crazy long error output when test fails in Travis build"

[crookedneighbor]

My hunch is the problem is in gulp.

[paglias]

When running some tests as standalone commands like gulp test:api-v3:integration there's no super long output... it's one more clue the problem is with gulp

[SabreCat]

I still see overlong output even when running individual tests that fail. Not _as_ long as otherwise, but still enough to scroll the actual failing tests out of buffer in my terminal session. I've been resorting to running tests with | tee testlog.txt and paging through that output to find what actually went wrong.

[khipkin]

The actual error in my super-long output from above was a simple syntax error, followed by many karma retries. I've had similar super-long TravisCI runs since then for various root causes, all with endless retries of karma.

Shouldn't the build skip all remaining checks once one fails? Why is it still trying after errors are detected?

[crookedneighbor]

It is not by design.

[crookedneighbor]

Oooh! I think this happens when the server crashes.

[khipkin]

Ohhhh good idea!!!

[SabreCat]

Would an ordinary test assertion failure crash the server? Because I get the zillion-line spew even when the site itself runs fine, if there's a failing test.

[librarianmage]

None of my tests seem to crash the server, but a few of them have run over an hour, and only the top 5% is useful information

[paglias]

Bumped priority to medium

[khipkin]

It seems like whenever a test fails, that test's test suite gets rerun recursively forever.

this build: (49 min 5 sec)

• after the test suite with the failed test:
  
  
  
  • npm ERR! Failed at the [email protected] test:common script 'mocha test/common --recursive'
  
  
• and then these messages over and over:
  
  
  
  • 'test' errored after 48 s
  
  
  • Error in plugin 'run-sequence(test:common)'
  
  

this build: (49 min 6 sec)

• after failed test:
  
  
  
  • Error: Command failed: /bin/sh -c NODE_ENV=test mocha test/api/v3/unit --recursive
  
  
• and then these messages over and over:
  
  
  
  • 'build:dev' errored after 2.35 min
  
  
  • Error in plugin 'run-sequence(test:api-v3:unit)'
  
  

this build: ()

• after failed test:
  
  
  
  • Error: Command failed: /bin/sh -c NODE_ENV=test mocha test/api/v3/integration --recursive
  
  
• and then these messages over and over:
  
  
  
  • 'test' errored after 7 min
  
  
  • Error in plugin 'run-sequence(test:api-v3:integration)'
  
  

[khipkin]

However, if the build fails because of linting errors, it just stops and says the build failed:

this build: (5 min 6 sec)

• npm ERR! Failed at the [email protected] lint script 'eslint .'.

this build: (5 min 11 sec)

• npm ERR! Failed at the [email protected] lint script 'eslint .'.

this build: (6 min 11 sec)

• npm ERR! Failed at the [email protected] lint script 'eslint .'.