Dredd: Verify how Dredd deals with non-2xx responses in OpenAPI 3

[honzajavorek]

Dredd skips non-2xx responses for OpenAPI 2:

• https://dredd.org/en/latest/how-it-works.html#id12
• https://dredd.org/en/latest/how-to-guides.html#id13

A similar behavior should be implemented for OpenAPI 3 as well. Currently the code only deals with OpenAPI 2.

[artem-zakharchenko]

Do I get it correctly that it's about extending this conditional skipping in TransactionRunner?

https://github.com/apiaryio/dredd/blob/f7e5ee45bccd1097a65c68884cfefff250a859ca/packages/dredd/lib/TransactionRunner.js#L344-L355

Can we assert that the API description document is OpenAPI 3 by checking:

openapi: "3.0.0"

We'd need to see what toTransactions returns:

https://github.com/apiaryio/dredd/blob/f7e5ee45bccd1097a65c68884cfefff250a859ca/packages/dredd/lib/Dredd.js#L251

[honzajavorek]

I'd say the steps are:

1. Learn about OpenAPI 3 and see whether the way it generates transactions is similar enough to OpenAPI 2 so that it makes sense to approach the underlying problem the same way.
2. If yes, then write tests verifying the desired behavior.
3. Then the media type in transaction.apiDescription.mediaType should give you relevant media type for the OpenAPI 3 (a different one, not containing swagger, that's why it probably doesn't work correctly now). See the OAS3 adapter code.

The underlying problem this tries to solve is that OpenAPI 2 generates transactions in big amounts (especially through produces and consumes) and a lot of them might be for 4xx status codes. People usually don't want to test those, or in other words, they usually cannot be tested out of the box, without hooks. That's why Dredd skips non-2xx transactions by default and lets users to unskip them in hooks. That's as close to the behavior most of the people would expect out of the box as possible.

[hedleysmith]

Dredd seems to be working fairly well for OAS3 for me, except for this issue (so far). I've started a PR which gets things working for OAS3: https://github.com/apiaryio/dredd/pull/1885