Xud: simulation tests: cannot start xud network

[erkarl]

I'm experiencing a possible race condition error in our simulation tests that causes the build to occasionally fail with the following output:

xud_test.go:92: xud process (4-Dave) did not shutdown gracefully. process (13835) killed
harnessTest.go:64: Error outside of test: *errors.errorString cannot start xud network: credentials: failed to append certificates
    /home/travis/build/ExchangeUnion/xud/test/simulation/xud_test.go:251 (0xa2df53)
      TestExchangeUnionDaemon: ht.Fatalf("cannot start xud network: %v", err)
    /home/travis/.gimme/versions/go1.12.linux.amd64/src/testing/testing.go:865 (0x4f96a0)
      tRunner: fn(t)
    /home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:1337 (0x45d371)
      goexit: BYTE  $0x90   // NOP

[moshababo]

I've seen this on Travis, but while trying to debug it the problem stopped. Didn't manage to figure out the reason for it though.
It looked like xud was getting stuck while trying to generate the certificate file (after the file was already created).

[kilrau]

can you reliably reproduce that? @erkarl

[erkarl]

@kilrau no, that's what makes this difficult to debug and occasionally makes the simulation tests fail. I consider this a low priority, but it would be nice to have simulation tests produce consistent results.

[kilrau]

I consider this a low priority, but it would be nice to have simulation tests produce consistent results.

Agree, that's why moved to post-1.0.0

[kilrau]

Did you see this too? @rsercano

[rsercano]

Yes, in fact this's the reason why we get automated build fails; #1142 but I really don't know where it comes from.

IMHO, this's a file read issue, if there's already a working build (sim-test) in VM, it opens the certificate file(s), if there's another build in the meantime, it tries to open it too, but since it's already opened by another build, it fails and throws the failed to append certificates error. But that's not a certain proved truth, that's just what I can think about it.

That's why I suggested to execute one build at a time (disabling push build & Node LTS build), I know it's not the best way but for now it could reduce the failure rate.

[rsercano]

I implemented a simple logic to try to read cert files of node 3 times before exiting with an error. Related PR: #1142

[kilrau]

I assume this is gone in the recent simulation test fixes, if not please reopen @erkarl