Shields: Daily Tests Job Failing (not service tests)

[calebcartwright]

We've had a handful of persistently failing service tests (seems like each time we've fixed one a new one popped up :smile:) that I hadn't even noticed the Run main tests step was failing and has been for at least a month.

0323034607 Error: I've made a huge mistake
    at ThrowingService.handle (/home/circleci/project/shields/core/base-service/base.spec.js:251:17)
    at Function.invoke (/home/circleci/project/shields/core/base-service/base.js:76:1710)
    at Context.<anonymous> (/home/circleci/project/shields/core/base-service/base.spec.js:255:31)
    at callFn (/home/circleci/project/shields/node_modules/mocha/lib/runnable.js:387:21)
    at Test.Runnable.run (/home/circleci/project/shields/node_modules/mocha/lib/runnable.js:379:7)
    at Runner.runTest (/home/circleci/project/shields/node_modules/mocha/lib/runner.js:535:10)
    at /home/circleci/project/shields/node_modules/mocha/lib/runner.js:653:12
    at next (/home/circleci/project/shields/node_modules/mocha/lib/runner.js:447:14)
    at /home/circleci/project/shields/node_modules/mocha/lib/runner.js:457:7
    at next (/home/circleci/project/shields/node_modules/mocha/lib/runner.js:362:14)
    at Immediate._onImmediate (/home/circleci/project/shields/node_modules/mocha/lib/runner.js:425:5)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)

https://circleci.com/gh/badges/daily-tests/1056

There's also other errors popping up in some of the logs, for example this one from a recent run:

0421034630 Error: Cannot push metrics. Cause: NetConnectNotAllowedError: Nock: Disallowed net connect for "shields-metrics.io:80/metrics"
    at InfluxMetrics.sendMetrics (/home/circleci/project/shields/core/server/influx-metrics.js:1:6674)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)

Haven't really had a chance to dig into it but wanted to open this to track it

[platan]

Errors regarding metrics was introduced recently. This is a fix https://github.com/badges/shields/pull/4931

[PyvesB]

We've had a handful of persistently failing service tests (seems like each time we've fixed one a new one popped up 😄)

Yes exactly! When I tackled #4061, only Code Climate services tests were failing, which was fixed by #4927. I was hoping we could get that badge green on our Readme! 😄

[calebcartwright]

Looks like the new Circle UI sheds a bet more light on the logs:

``bash TypeError: Cannot destructure propertyResolver` of 'undefined' or 'null'.

at Object.<anonymous> (/home/circleci/project/shields/node_modules/cacheable-lookup/source/index.js:9:5)

at Module._compile (module.js:653:30)

at Module.replacementCompile (/home/circleci/project/shields/node_modules/append-transform/index.js:60:13)

at Module._extensions..js (module.js:664:10)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/append-transform/index.js:64:4)

at Module.load (module.js:566:32)

at tryModuleLoad (module.js:506:12)

at Function.Module._load (module.js:498:3)

at Module.require (module.js:597:17)

at require (internal/module.js:11:18)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/got/dist/source/index.js:7:28)

at Module._compile (module.js:653:30)

at Module.replacementCompile (/home/circleci/project/shields/node_modules/append-transform/index.js:60:13)

at Module._extensions..js (module.js:664:10)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/append-transform/index.js:64:4)

at Module.load (module.js:566:32)

at tryModuleLoad (module.js:506:12)

at Function.Module._load (module.js:498:3)

at Module.require (module.js:597:17)

at require (internal/module.js:11:18)

at Object.<anonymous> (/home/circleci/project/shields/core/got-test-client.js:3:13)

at Module._compile (module.js:653:30)

at Module.replacementCompile (/home/circleci/project/shields/node_modules/append-transform/index.js:60:13)

at Module._extensions..js (module.js:664:10)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/append-transform/index.js:64:4)

at Module.load (module.js:566:32)

at tryModuleLoad (module.js:506:12)

at Function.Module._load (module.js:498:3)

at Module.require (module.js:597:17)

at require (internal/module.js:11:18)

at Object.<anonymous> (/home/circleci/project/shields/core/base-service/legacy-request-handler.spec.js:7:13)

at Module._compile (module.js:653:30)

at Module.replacementCompile (/home/circleci/project/shields/node_modules/append-transform/index.js:60:13)

at Module._extensions..js (module.js:664:10)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/append-transform/index.js:64:4)

at Module.load (module.js:566:32)

at tryModuleLoad (module.js:506:12)

at Function.Module._load (module.js:498:3)

at Module.require (module.js:597:17)

at require (internal/module.js:11:18)

at /home/circleci/project/shields/node_modules/mocha/lib/mocha.js:314:36

at Array.forEach (<anonymous>)

at Mocha.loadFiles (/home/circleci/project/shields/node_modules/mocha/lib/mocha.js:311:14)

at /home/circleci/project/shields/node_modules/mocha/lib/mocha.js:347:12

at new Promise (<anonymous>)

at Mocha.loadFilesAsync (/home/circleci/project/shields/node_modules/mocha/lib/mocha.js:346:12)

at singleRun (/home/circleci/project/shields/node_modules/mocha/lib/cli/run-helpers.js:107:15)

at exports.runMocha (/home/circleci/project/shields/node_modules/mocha/lib/cli/run-helpers.js:144:11)

at Object.exports.handler (/home/circleci/project/shields/node_modules/mocha/lib/cli/run.js:306:11)

at Object.runCommand (/home/circleci/project/shields/node_modules/mocha/node_modules/yargs/lib/command.js:242:26)

at Object.parseArgs [as _parseArgs] (/home/circleci/project/shields/node_modules/mocha/node_modules/yargs/yargs.js:1096:28)

at Object.parse (/home/circleci/project/shields/node_modules/mocha/node_modules/yargs/yargs.js:575:25)

at Object.exports.main (/home/circleci/project/shields/node_modules/mocha/lib/cli/cli.js:68:6)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/mocha/bin/mocha:133:29)

at Module._compile (module.js:653:30)

at Module.replacementCompile (/home/circleci/project/shields/node_modules/append-transform/index.js:60:13)

at Module._extensions..js (module.js:664:10)

at Object.<anonymous> (/home/circleci/project/shields/node_modules/append-transform/index.js:64:4)

at Module.load (module.js:566:32)

at tryModuleLoad (module.js:506:12)

at Function.Module._load (module.js:498:3)

at Function.Module.runMain (module.js:694:10)

at startup (bootstrap_node.js:204:16)

at bootstrap_node.js:625:3

null

undefined

7

foo

4.25
```

[calebcartwright]

Looks like the Services have unique names test is timing out/failing consistently in the daily jobs run

[calebcartwright]

:disappointed: it looks the test is still timing out even with the 5 second timeout

https://app.circleci.com/pipelines/github/badges/daily-tests/198/workflows/326ad978-a733-46bb-9dc3-68e57b79955d/jobs/1094

```bash
1035 passing (10s)

1 failing

1) Services have unique names:

 Error: Timeout of 5000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/circleci/project/shields/services/check-services.spec.js)
  at processImmediate (internal/timers.js:456:21)
  at process.topLevelDomainCallback (domain.js:137:15)

```

[paulmelnikow]

I guess that test imports the whole server… which probably also means instrumenting the whole server. Though I'm still surprised that would take more than 5 seconds!

[calebcartwright]

I guess that test imports the whole server… which probably also means instrumenting the whole server. Though I'm still surprised that would take more than 5 seconds!

:relieved: it's working again with the increased timeout. now that the test is running through completion, it looks like it's taking just north of 5 seconds in most cases so we could probably cut it down to 10 seconds at some point in the future

[calebcartwright]

Closing this as the Main tests phase is passing again :tada: