Athens: Split up our tests (unit, integration, end to end)

[arschles]

Following https://github.com/gomods/athens/pull/256#issuecomment-404931246 and I believe there was discussion on this in yesterday's meeting (7/12/2018).

Here's my opinion on how we should split up our tests:

1. Unit tests - only operate on in-memory things
2. Integration tests - operate on services that a developer can run on their machine via docker compose
3. Full end to end - operate on everything including live cloud services

(1) is easy for anyone to run and we have pretty good docs (IMO, not sure if peeps agree?) for how to set up your environment for (2). (3) is just starting to become a thing in @robjloranger's https://github.com/gomods/athens/pull/208. I think he added a switch in the code to turn the live GCP tests on or off, so that might be a good thing to build on top of.

[michalpristas]

I totally agree on that, maybe we can talk about 3. I would like to see how it works in k8s.
in my current org we have real services, isolated storages... which ofc cost money.

[marwan-at-work]

For my dev set up I'd like two things:

1. Have one command to bootstrap only things I need to run in dev mode. At this point: it's just redis, mongo, jaegar and the registry/proxy servers. And another command to tear down everything and remove persistent data (docker volume prune)

1. Have a clean slate backdoor to not have to take down everything and reboot everything once again just so that we have a clean cache/storage.

@arschles this issue may not be as related to move the comment above here, let me know if it should be another issue

[arschles]

@marwan-at-work I feel like our test code and test/dev dependencies are related enough to keep in the same issue for now. Would you be up for PR-ing the command to start up things only needed in dev mode? That'd be a good start down this path.

[marpio]

I was thinking, maybe we could use minio for almost everything. We already have storage backed by minio. It is s3 compatible so in theory we could use it a CDN storage for the registry (not sure if aws-sdk can be used to talk to minio though). The only thing that's missing is the event log impl. on top of minio (at the moment mongo only). Then we would only need minio, redis and jaeger for proxy and registry development. What do you think?

[arschles]

I like it. We still have rdbms and mongo storages as well - are you
thinking about something else for those?

On Tue, Jul 24, 2018 at 04:43 marpio [e-mail] wrote:

I was thinking, maybe we could use minio for almost everything. We already
have storage backed by minio. It is s3 compatible so in theory we could use
it a CDN storage for the registry (not sure if aws-sdk can be used to talk
to minio though). The only thing that's missing is the event log impl. on
top of minio (at the moment mongo only). Then we would only need minio,
redis and jaeger for proxy and registry development. What do you think?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/262#issuecomment-407346966, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEU0d9birwq6NLsq4WJvfYM5Mi45javks5uJuw7gaJpZM4VPZ4x
.

[marpio]

I was actually only thinking about the development setup @arschles sorry for not being clear on that.
It would be nice if we would only need minio, jeager and redis for the proxy and registry development env. What do you think @marwan-at-work ?
We still need the others dependencies for tests.

[arschles]

@marpio that makes sense. What do you think about making unit tests completely free of outside dependencies? For example, #343 is removing all dependencies on external GCP, we could do the same thing for RDBMS, Jaeger, and so on. I'm not sure if this is wise but I want to hear what you think

[marpio]

@arschles I like the idea. I would love to have a set of tests which can be executed often while I change the code and gives me at least some degree of confidence that I didn't break anything important. This should be fast so removing external dependencies is important.
Besides that, mocking the dependencies would allow us i.e. to easily test error conditions.
I'm gonna go through the components which use outside deps and create an issue for each.

[marpio]

@arschles I'm thinking, should we close this and open a new more specific issue?

We can already run tests without any additional services. We also have integration tests using minio and mongo.
Should we still have tests that operate on live services?
s3/DO spaces can be potentially replaced with minio and Azure Blob Store with Azurite, we don't have anything for GCP though.

[arschles]

Note that https://github.com/gomods/athens/pull/888 is open, and when merged it provides a docker compose target that lets you start up an athens locally with "things" attached to it.

@manugupt1 mentioned in the Nov. 29 meeting that we could have multiple docker compose targets to start Athens in dev and also to start Athens for production.

I think we can use this type of setup to run different levels of integration tests

@marpio how do you feel about using the "integration test" docker compose targets to do our tests against Azurite/Minio (and mongo for data storage), and not running tests on live services (i.e. no GCP testing unless there's an emulator out there for it - I don't know of one either)?

If you're cool with that, we can probably keep this issue open and change the title.

Thoughts?

[marpio]

@arschles :+1: yes, that was my thinking as well. We can cover most of the storages by using things like minio, azurite etc.
I couldn't find anything for gcp either.

[arschles]

On the Dec. 6 Meeting, we talked about this:

• @manugupt1 suggested that we document what storages folks actually use in "production"
  
  
  
  • To help set the expectation on what's been tested in the wild
  
  
• We create another docker-compose service called test that starts up every service that we're able to start in a docker container, and runs all our tests against them
  
  
  
  • We have a dev target that does the same thing for running the server in https://github.com/gomods/athens/pull/888/files
  
  

[arschles]

I am going to take this out of 0.3.0 because it has been progressing in bits and pieces. I'll break this out and reference the individual pieces of work

[arschles]

This is very broad and I'm going to close it. We have several specific issues & PRs in progress right now that are splitting up and expanding our tests.