Xud: Simulation tests dev roadmap - beta.4

[LePremierHomme]

This issue is to track the simulation tests dev roadmap and discussions. Tasks are ordered by priority:

• [x] dockerize the simulation tests
• [x] support npm run test:sim:logs
• [x] support npm run test:sim:run:security
• [x] support npm run test:sim:run:instability
• [x] support faster test code builds
• [x] update README.md and provide npm scripts for test developer
• [x] before allowing RPC calls, verify that xud has finished the initialization phase (not returning "xud is starting" error)
• [x] when custom-xud is being downloaded/installed, extend the test context lifetime
• [x] cache custom-xud downloads
• [x] debugging: export lnd logs

• [x] security/coverage: https://github.com/ExchangeUnion/xud/issues/1126

  - [ ] coverage: https://github.com/ExchangeUnion/xud/issues/1016

• [x] fix TestSecurityUnsettledChannels/maker_shutdown_after_1st_htlc

• [x] fix TestInstability/maker_crashed_after_send_payment_with_delayed_settlement

  - [x] run above two on travis: https://github.com/ExchangeUnion/xud/issues/1220

• [ ] test infra stability: https://github.com/ExchangeUnion/xud/issues/1315

[erkarl]

For dockerizing the test environment a good first step would be to create a geth container from these scripts:
https://github.com/ExchangeUnion/xud/blob/master/test/simulation/install.sh#L6-L11

https://github.com/ExchangeUnion/xud/blob/master/test/simulation/install.sh#L9 should go into a separate raiden container.

[kilrau]

Regarding our discussion whether to integrate xud-docker into the dockerized similuation tests: let's hold off on this. While it will be somehow possible, I'm afraid that it'll set us back for quite some time. Getting e.g. geth and contracts dockerized and tests stable is prio #1.

Could you open a WIP PR so we can track status? @LePremierHomme

[kilrau]

I want to add one requirement though: it has to be easily possible to run the tests not only with different xud versions, but also the same way with different raiden & lnd versions. The raiden team should be able to use our tests for developing raiden the same way we use our tests for developing xud.

[kilrau]

As discussed on discord, we should also consider using bitcoind & litecoind regtest to replace btcd and especially ltcd. You can add this to the requirement list above @LePremierHomme

[sangaman]

I was about to create a separate issue for the test:sim:logs script but I see it's covered here. It's broken currently and it would be useful for debugging some of the other flaky issues with the simulation tests - I ran into that limitation just now - so it would be good to get that working soon if possible.

[kilrau]

Added https://github.com/ExchangeUnion/xud/issues/1057

[kilrau]

Added https://github.com/ExchangeUnion/xud/issues/1126

[kilrau]

Please close https://github.com/ExchangeUnion/xud/issues/1220 if all tests are in, add if not @LePremierHomme

[LePremierHomme]

Closing.
Regarding the 2 last remaining tasks:

• #1016 - lower priority for now, we'll be taken care of later.
• #1315 - unreproducible, might have already been solved after the migration to docker.