Dredd: CLI `--only` doesn't skip hooks for other transactions

[ddelnano]

I am trying to run a single transaction since running the entire suite can be time consuming. I tried using the --only cli flag but that runs each transaction as if each one was skipped, which takes just as much time as running them. Is there a way to accomplish what I am looking for? If not would a contribution that does what I am looking for be accepted?

[ddelnano]

I tried doing something like this when using the --only flag.

However, the transactions passed to each of my beforeEach hooks has the skip property set to false, which does not allow me to prevent the hooks from running its time consuming operation.

[ddelnano]

I took a look at the code and noticed that the only option is checked after both the beforeEach and before hooks are run and it decides to not run it just before it makes the request. So it sets the skip property, but at a much later point in execution.

[honzajavorek]

I think the problem might be, as you more or less pointed out, that the implementation of --only is done by skipping. skip is just an attribute of the internal transaction object and can be changed by hooks, so it won't prevent hooks to run. While for --only it doesn't make much sense to have the ability to unskip some of the skipped transactions, there are other use cases when it makes some sense (e.g. newly added http://dredd.readthedocs.io/en/latest/how-to-guides/#testing-non-2xx-responses-with-swagger), so the fact that skipped transactions go to hooks is desirable behavior. And I think there is no standard thing/flag yet in Dredd to completely prevent transaction to get handled by hooks.

I'm also not sure whether we should focus on this rather than on the original cause of your problem - Dredd testing with hooks can be slow. I think there is quite a space for speed optimizations.

[ddelnano]

@honzajavorek that makes sense. So in order to understand why I am putting up with this expensive hooks step I am going to detail how I am using Dredd differently in this situation.

So previously when I had used Dredd, in the hookfiles I would insert records into the DB and reset the data. However, what I am trying to do now is use Dredd to not only test the API but also force the devs on this project to maintain a fully complete data set for the application. So essentially all dredd does in the hooks is start and stop docker containers and when the database container starts up it sucks in a sql dump file that has all the test data. This is awesome because our api tests can't pass unless we have updated our data set. The purpose of keeping this dataset current is that this dataset will be used to integration test across all of our micro services. So basically we will be using a container orchestration tool to create an ephemeral environment to launch all pieces of our infrastructure and each microservice will be using a database container that uses the test data set that I have been describing. So it allows our API docs to stay up to date and our integration tests to be current.

So there are two reasons why I would love this feature. For running dredd locally, it really only makes sense for our developers to run a few (if not just one) dredd transaction that is related to the feature they are working on because our CI pipeline will run all of them. However, there is also massive benefit for doing this in CI as well. Imagine I have a apiblueprint file with 30 transactions and my application is two containers: web app and database. The way our dredd is run in CI is it is a docker container that can run docker-compose applications. So if I wanted to, I could launch each dredd container specifying to only run 1/3 of the transactions each and they in term would spawn their own instance of the application and I could have parallel dredd tests since docker containers are so lightweight. So in this example, I could launch 3 dredd containers, which would spawn 2 more containers each (for a total of 9 containers) with 10 transactions and cut the Dredd test runtime to 1/3 in theory.

[netmilk]

@ddelnano @honzajavorek This is caused by how the skipping works internally. It just skips the actual HTTP call and it's validation - it's more like a dry-run then actual skip. I'm not sure if we should not call the hooks for skipped transactions at all because you may want to un-skip the programmatically in hooks.

I'd vote to change the implementation of --only and filter the transactions before it's passed to the transactinoRunner and not to change the actual implementation of transactionRunnner

[ddelnano]

@netmilk I see you labeled this as a bug? So does that mean if I were to give implementation a shot it would be accepted? From @honzajavorek 's previous comments it seemed to me that its current behavior was desired.

[netmilk]

@ddelnano lol - you're blazing fast.

I think it's totally worth redoing and your contribution will be more than welcome! :) I think it's a bigger project to make it right but we can kick it off!

What I propose:

I would pull code from transactionRunner implementing:

• --sorted
• --only
• --method
• --names
• --headers

and move all this functionality to a new transactionsPostprocessor which will be hooked in Dredd#run after compileTransacions.

This transaction postprocessor will have functions with the same interface like beforeAll hooks called conditionally based on the Dredd options. This postprocessor will take care of removing or modifying the transactions in the compiled runtime (transactions variable on Dredd instance).

@ddelnano Please let me know if it makes sense to you.

[ddelnano]

haha

@netmilk yea all of that sounds good to me. I'll have time this weekend to start digging into it.

[benneely]

I'm also interested in finding a fix for this issue. Another potential solution would be to have --only set the skip attribute of the internal transaction object to true at an earlier point in the Dredd lifecycle - before all transactions. Each dredd hook handler could then look to this attribute before executing the intended hook. Admittedly this is a little more hacky than the proposed solution above, but seems like it might be an easier implementation and also allow more flexibility (overwriting --only in the hooks).

[ddelnano]

@benneely I started working on this and got a first pass at the post processor done. Didn't get to try integrating it into dredd. I am going on vacation until Monday but will work on it once I get back. I will try to update this issue with my progress and push some stuff once I get closer.

[benneely]

sounds good and thanks!

[honzajavorek]

Wow, great to see such activity here! I'd be glad to help if guidance is needed. However, I'm not really sure this is a "bug", I think it's rather an improvement of current state of things. I'll re-label and discuss with @netmilk eventually.

[iamareebjamal]

Any updates? This makes quick and granular testing of endpoints very hard

[ddelnano]

I never got a full implementation of this working for this.

[McKean]

@ddelnano is there a plan to get this working or is this open for anyone to pick up the bat?

[ddelnano]

I have no plans to work on this so feel free to pick it up