Rh-che: Investigate strange intermittent test fail building wsagent

[ScrewTSW]

Issue problem:
Master build tests fail with a strange error in one of unit tests:

[ERROR] Tests run: 31, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.925 s <<< FAILURE! - in TestSuite
[ERROR] sendWorkspaceStoppedOnDestroy(com.redhat.che.plugin.analytics.wsagent.AnalyticsManagerTest)  Time elapsed: 0.033 s  <<< FAILURE!
java.lang.AssertionError: null: lists don't have the same size expected [2] but found [3]
    at com.redhat.che.plugin.analytics.wsagent.AnalyticsManagerTest.sendWorkspaceStoppedOnDestroy(AnalyticsManagerTest.java:513)

https://ci.centos.org/view/Devtools/job/devtools-rh-che-build-che-credentials-master/119/console

[ERROR] Tests run: 33, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.653 s <<< FAILURE! - in TestSuite
[ERROR] sendWorkspaceUsedOnActivityAfterInactivity(com.redhat.che.plugin.analytics.wsagent.AnalyticsManagerTest)  Time elapsed: 0.028 s  <<< FAILURE!
java.lang.AssertionError: Lists differ at element [0]: Keep Workspace Inactive in Che != Use Workspace in Che expected [Keep Workspace Inactive in Che] but found [Use Workspace in Che]
    at com.redhat.che.plugin.analytics.wsagent.AnalyticsManagerTest.sendWorkspaceUsedOnActivityAfterInactivity(AnalyticsManagerTest.java:467)

https://ci.centos.org/view/Devtools/job/devtools-rh-che-build-che-credentials-master/279/console

version: 6.11.1

• [ ] I can reproduce it on latest official image

Reproduction Steps:

Intermittent issue with build.

Runtime:

runtime used:

• [ ] minishift (include output of minishift version)
• [ ] OpenShift.io
• [ ] Openshift Container Platform (include output of oc version)
• [ ] k8s (include output of kubectl version)

[ibuziuk]

More logs from https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/779/console

15:55:58.837 [main] DEBUG com.redhat.che.plugin.analytics.wsagent.AnalyticsManager - sending Use Workspace in Che (ip=theIp - userAgent=theUserAgent) with properties: {}
15:55:58.838 [main] DEBUG com.redhat.che.plugin.analytics.wsagent.AnalyticsManager - scheduling ping request with the following delay: 5
15:55:58.849 [main] DEBUG com.redhat.che.plugin.analytics.wsagent.AnalyticsManager - CheckActivityPeriod: 5
[ERROR] Tests run: 33, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.824 s <<< FAILURE! - in TestSuite
[ERROR] sendWorkspaceStoppedOnDestroy(com.redhat.che.plugin.analytics.wsagent.AnalyticsManagerTest)  Time elapsed: 0.023 s  <<< FAILURE!
java.lang.AssertionError: null: lists don't have the same size expected [2] but found [3]
    at com.redhat.che.plugin.analytics.wsagent.AnalyticsManagerTest.sendWorkspaceStoppedOnDestroy(AnalyticsManagerTest.java:517)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   AnalyticsManagerTest.sendWorkspaceStoppedOnDestroy:517 null: lists don't have the same size expected [2] but found [3]

[ibuziuk]

the test is continue to be unstable and we might consider disabling it

[ibuziuk]

PR is sent - https://github.com/redhat-developer/rh-che/pull/1268

[ibuziuk]

PR is merged. Closing

[ibuziuk]

re-opening, looks like it is still failing - https://ci.centos.org/view/Devtools/job/devtools-rh-che-build-che-credentials-master/273/console

[amisevsk]

Looks like a similar but slightly different problem, I'll take a look.

[amisevsk]

From what I've been able to find:

• Issue seems to be due to an NPE in sending a ping request, which only occasionally happens
  
  
  
  • Issues with e.g. the ping seem to affect other tests / how we track events needs to maybe be improved
  
  
• I'm not able to reproduce the exact failure yet but I'm able to generate similar failures by deliberately causing an NPE while debugging tests
• There's probably also some interplay from the CI machine, where slowness / something else is affecting the tests

To fix this, some refactoring would need to be done to the analytics manager to make it more modular or avoid these issues (currently, part of the problem is that EventDispatcher is an internal class in AnalyticsManager which makes mocking more difficult. We discussed instead during standup that we should consider disabling the problem tests for now, since this feature 1) fails on CI only, and only occasionally and 2) will need to be worked on later to make it work with Che 7 workspaces anyways.

[ibuziuk]

Temporary ignoring 'sendWorkspaceUsedOnActivityAfterInactivity' test - https://github.com/redhat-developer/rh-che/pull/1284

[garagatyi]

What is the plan with this issue? Are we going to postpone finding the solution or we want to take it into the next sprint and work on it again?

[amisevsk]

My understanding is that the test fixes + refactor involved are prioritized relatively low since analytics needs to be reworked in general for Che 7 workspaces.

I took a look at refactoring analytics manager recently and it was not entirely straightforward.

[ibuziuk]

we are going to postpone the work related to fixing ignored tests, since those might be outdated fairly soon with telemetry for Che 7

[ibuziuk]

wsagent was removed as part of Che 7 GA transition from master branch. Closing