Origin: Basic Integration Testing

[DanielVF]

Currently Origin.js is decently tested, and those tests checked at every PR. This really helps our velocity when working with it - you can be reasonably sure that when you start work, things won't be broken, and reasonably sure that when the tests pass, your PR won't break anything inside origin.js

Unfortunately, we do not have these same guarantees that the rest of our tools will be working or working with each other. At any given moment, a change to one system could break others, and not everything in the box may start up when you bring it up. I spent the last week working outside the origin.js comfort-zone and I've have seen the evil that lurks there. :D

I think even an incredibly basic set of integration tests would be massive help.

All we really need to start with just the success path, using the services that the box provides:

• Add a listing using origin.js and make an offer on it.
• Verify that the listing can be retrieved using GraphQL from the discovery server.
• Verify that the listing can be query searched from the discovery server.
• Verify that the notification server makes a notification call.

If we test just these two, we'll have verified the basic setup and liveness of:

• Postgres
• Elasticsearch
• local IPFS
• local blockchain
• origin-discovery
• event-listener
• origin-notifications
• all of the above interacting with origin.js

This will get us quite aways farther towards keeping master in good shape, and allowing new contributors to get started without a few hours of debugging their environment or an currently broken services.

The actual tests are easy. I think the harder part is how we should run the these tests. While ideally we'd just use the box, that may be test time prohibitive?

Thoughts? Best way to actually run something this as a part of every PR?

@tomlinton @franckc @sparrowDom @micahalcorn

[tomlinton]

I've been doing some thinking/work towards this, but it is is tricky. The options I see right now are:

1. Create a Kubernetes cluster using Minikube as part of the CI process and deploy the Origin stack using the Helm chart we have. Advantages: as close as we'll get to testing in our deployment environment. Disadvantage: slow, will need extra stuff that isn't part of our standard deployment.
2. Create a test stack using Docker Compose. Advantages: reasonably close to what we'll get in a deployment environment, also a test that origin-box is functioning properly. Disadvantage: slow.
3. Straight JavaScript tests. Advantage: should be fast. Disadvantage: unsure how to handle things like Elasticsearch.

Any of these options can be used as part of TravisCI. But it is tricky working within the constraints so there is an option to roll our own test setup in the same way I did the deployments. I did some work to make 1. a possibility by separating our stuff from all the Ethereum services we have, but I think I'm leaning towards 2.

I think I'll probably work up a prototype with Docker Compose and see how it goes.

[franckc]

Yeah, I think option 2 might be the simplest to start with.

I could envision a few integration tests at origin-js layer could go a long way validating the origin-js + discovery + ipfs + blockchain stack. For example

• create a listing, query disco to make sure it is indexed
• make offer on listing, query disco

Now for some other components it would be more tricky to write integration tests

• DApp: hard because it is front-end code
• origin-notification: hard because it involves FE subscription flow. though we could test the listener -> notification server integration

[sparrowDom]

I like the option 2 approach best as well. Ideally this stack would be configured in such a manner that it could run alongside origin-box and have no resource/port collisions. This way the slowness would have lesser impact and developers wouldn't need to shutdown origin-box to spin up testing stack.

Just thinking what sort of tools we would need to develop in order to get option 2 running:

• docker configuration - to get everything running (probably the hardest part)
• clear state scripts - something used by beforeEach of every integration test that resets the state of the db, elasticsearch, resets the current block of the listener(?), clears ganache state.
• utility scripts used by the tests:
  
  
  
  • wait for event listener to index the transaction with timeout that fails the test
  
  
  • (probably more I can't think of ATM)
  
  

[DanielVF]

@franckc On origin-notifications, the browser push messages are just HTTP(s) calls. If we spin up a local server as a part of the test, and give that server as the address when creating the push subscription, then we can test the notifications without making changes to the notification server.

[DanielVF]

@sparrowDom We might be able to write the integration tests in such a way that they don't require a clean system.

[tomlinton]

Another idea I had was to rename the dev deployment we have to test. When something gets merged into master it gets deployed here, then tests are run against the live deployment. If the tests pass whatever the code is gets promoted to staging. This might be an option if we find it is too slow to run these tests on every push/PR.

[sparrowDom]

@danielVf imho it is kind of nice to have a test environment where state/execution of one test does not effect the execution of another, though it is probably a really low priority thing. (since this has not been a problem so far afaik)

After doing some research it seems that it is not straightforward to reset state in Ganache quickly (which would be needed in case we would decide to reset state before each test). Leaving a link with a possible solution here in case it is needed in the future - reset state in Ganache tests.

[DanielVF]

A big advantage of not needing to reset state between tests is that we could run the same test suite against a staging cluster.

[tomlinton]

I think you could revert the state using the --db option. I briefly played around with that when I was thinking about having the state of the blockchain baked into origin-box rather than creating it every time it starts.

[sparrowDom]

Besides db option there are also evm_snapshot and evm_revert commands to consider. To see what serves our purpose best.

On the first glance it seems that db option preserves the state between different Ganache executions while snapshot/revert lets you load/save state within the same run.

[tomlinton]

I think we can probably close this, there is more work to do on the integration tests but the basic system is in place. We can raise more specific issues for further work.