Realworld: Backend REST API Testing - Chakram/Newman/Frisby?

[anishkny]

In the past I have had fun writing automated tests for REST APIs using Chakram. The provided Postman files are great while developing, but it may be worth having automated REST API tests.

Maybe we can "auto-derive" Chakram tests from the Postman specs or have both automated tests and Postman specs derived from a single source, to achieve a one-truth model for the API spec.

Alternately, Newman seems promising as an automated Postman collection runner.

Another library that can be used for this is FrisbyJS.

[lolripgg]

Can you clarify what problem we're trying to solve here? What purpose does automated tests provide that the Postman tests do not?

[EricSimons]

I think @anishkny is looking for a more rigid way of testing the API endpoints, which I think is fair. Our postman collection could certainly use more work/polish and beyond that it doesn't provide a more _real_ TDD approach that's designed to actually try and break your backend.

Another key insight brought up here is that we could create "export to [your ideal testing tool]" by doing some simple wrappers around the Postman collection. No idea how difficult that would be to do, but I'm all for enabling RW devs to use their favorite tools while building the apps 👍

[Cameron-C-Chapman]

So I like this idea as well and I started playing around with Newman to see how useful this could be and how far off from being useful now it was and I actually made some pretty good progress (because it wasn't really much work).

So I pushed a branch called add-postman-newman-cli-runner, I haven't issued a pr yet, I kind of wanted everyones thoughts first.

So if you pull down that branch and

1. Install newman npm install -g newman
2. cd into the api folder and run the follwing command newman run Conduit.postman_collection.json -e Conduit.postman_integration_test_environment.json

you'll see the collection tests being tested. :-)

I added an integration test specific environment file, updated a few tests that are failing now to only run in the normal environment, and moved the Auth tests to the top of the collection so they are run first.

The win here is that this would allow us to create a Travis CI build and have a requirement that the build passed on any collection updates before they are merged to master.

Let me know what you think and I can create a PR.

[EricSimons]

Wow, this is _awesome_! I think we should def implement this 👍 Thoughts @gothinkster/realworld-admins?

[sandeesh]

This is a great idea. I had a similar plan to help automate testing for all backends. Although I think we need to baseline the current spec and test before we can start implementing this with all the backends. The current spec needs proper response codes and standardised error messages along with validation criteria. Currently these things are being mixed up with different backends since there's no guidelines set in the spec.

[Cameron-C-Chapman]

I was thinking of this more as a way to automate the test of the realworld spec itself as it evolves, not necessarily for a developer to use during development. They have the postman collection and now there is a swagger spec they can use. The thought here (at least to me anyway) is that as we update the spec/postman collection, for example updating a response code, as that pull request is issued to master, the automated test could run and we would know right away that either a test needs to be written, or the change broke the current spec/collection.

It's not 100% now but I think even in it's current state it would help us validate any changes to the collection without having to force all the current projects to implement every change immediately. If they wanted to have a Travis CI build automated and/or always keep in sync with the realworld master they could, but the realworld master automated build doesn't necessarily have to affect every downstream project immediately.

At least that's the way I drew it up in my head :-)

[EricSimons]

Agree with both @Cameron-C-Chapman and @SandeeshS — we def need a new detailed spec, and the Newman tests can/should be upgraded to match once it's ready 👍

I just messaged @SandeeshS on gitter to see if I can help at all with getting the new spec done & I just merged @Cameron-C-Chapman's PR into the repo 💯 Awesome work!