Dredd: Support for testing over an https proxy?

[erajkovic]

Describe your problem

Support for HTTPS / with proxy ?

From reading the doc, after finding the question to run with debug, looks as if the answer was no - would be nice to have this upfront as an english message in the tool.

"Dredd intentionally does not support HTTP(S) proxies for testing. Proxy can deliberately modify requests and responses or to behave in a very different way then the server under test. Testing over a proxy is, in the first place, testing of the proxy itself. That makes the test results irrelevant (and hard to debug)."

For corporate users, it may be good to have an option which enable proxy.

What command line options do you use?

$dredd https://apicatalog.oraclecloud.com/public/v1/orgs/oracle-public/apicollections/apicatalog/1.0/apis/Categories/canonical https://apicatalog.oraclecloud.com

error: 
Error when loading file from URL 'https://apicatalog.oraclecloud.com/public/v1/orgs/oracle-public/apicollections/apicatalog/1.0/apis/Categories/canonical'.
Is the provided URL correct?

What is in your dredd.yml?

none

What's your dredd --version output?

dredd v5.1.6 (Darwin 17.4.0; x64)

Does dredd --level=debug uncover something?

verbose:
HTTP(S) proxy specified by environment variables: NO_PROXY=localhost,127.0.0.1,erajkovi-mac,.oracle.com,.oraclecorp.com,/var/run/docker.sock, http_proxy=http://www-proxy.us.oracle.com:80, HTTPS_PROXY=http://www-proxy.us.oracle.com:80, https_proxy=http://www-proxy.us.oracle.com:80, no_proxy=localhost,127.0.0.1,erajkovi-mac,.oracle.com,.oraclecorp.com,/var/run/docker.sock, HTTP_PROXY=http://www-proxy.us.oracle.com:80. Please read documentation on how
Dredd works with proxies:
https://dredd.readthedocs.io/en/latest/how-it-works/#using-https-proxy

debug: Downloading https://apicatalog.oraclecloud.com/public/v1/orgs/oracle-public/apicollections/apicatalog/1.0/apis/Categories/canonical errored: Error: tunneling socket could not be established, cause=getaddrinfo ENOTFOUND www-proxy.us.oracle.com www-proxy.us.oracle.com:80

With proxy active on corporate network, getting another message:

warn: Parser warning in file 'https://apicatalog.oraclecloud.com/public/v1/orgs/oracle-public/apicollections/apicatalog/1.0/apis/Categories/canonical':
Could not recognize API description format.
Falling back to API Blueprint by default.
on line 1
warn: Parser warning in file 'https://apicatalog.oraclecloud.com/public/v1/orgs/oracle-public/apicollections/apicatalog/1.0/apis/Categories/canonical': ignoring possible metadata, expected ' : ', one one per line on line 1

Can you send us failing test in a Pull Request?

[honzajavorek]

You are right, Dredd does not support proxy for _testing_. It supports proxy for reporting, i.e. --reporter=apiary, and fetching the API description document from URL address, i.e. dredd http://example.com/apiary.apib http://localhost:3000.

For corporate users, it may be good to have an option which enable proxy.

I understand corporate users often live behind a proxy, but the paragraph you quoted still holds.

When you test through proxy, you are testing the proxy instead of the API, and the test results become irrelevant. Imagine you have some headers in your specification, Dredd sends them, but the proxy strips them on the way. The request fails, although both Dredd and your API are fine. False negative, very hard to debug.

Worse example - imagine the proxy adds something, which accidentally makes your API compliant with the spec. Tests pass and you may think everything is fine. You deploy the API to production and make it public, with no proxy, and it won't work as intended. False positive. The whole purpose of Dredd gets beaten.

I'm fine leaving this issue open - let's discuss the topic, I can be missing something. But I'm yet to hear about a good way how to overcome the problem mentioned in the documentation.

[opichals]

@honzajavorek While acknowledging the dangers of through-proxy API testing I agree that this would indeed be a useful thing to have for people that have no other option but to use the proxy.

One could use something like this but that adds to the layers that may interfere.

[erajkovic]

I'll have to run some more tests to figure out if the issue was a user error or a bad configuration while using proxy over VPN - this morning, on the corporate network, I am able to get a local run pushed to apiary and see it under https://app.apiary.io/apicatalog/tests/runs#local.
Also, did not realize the 'danger' of using Dredd against a remote instance - in the summary below, one of the 'pass' was a DELETE call, but only on the first run ;)
complete: 10 passing, 24 failing, 0 errors, 14 skipped, 48 total
complete: Tests took 1762ms

[opichals]

@honzajavorek There are also situations where the remote API is going to be accessed through a proxy in production. In such a case the through-proxy testing is actually exactly what you need to be doing as it is the way things are deployed.

[honzajavorek]

I am willing to think about a solution where the user is allowed to use proxy for testing, but Dredd's output shows a clear warning/disclaimer that using the proxy means testing the proxy itself, not the target API. Then it's up to the user whether they accept the risks or whether they know what they're doing.

[honzajavorek]

@erajkovic I'm glad you could get it working! And yes, beware running Dredd against live API instances 🙂

[honzajavorek]

 I am willing to think about a solution where the user is allowed to use proxy for testing, but Dredd's output shows a clear warning/disclaimer that using the proxy means testing the proxy itself, not the target API. Then it's up to the user whether they accept the risks or whether they know what they're doing.

I propose this as a solution to the problem. This isn't a priority issue though - I invite everyone interested in getting this implemented to add reaction to the original post so your desire to use this is visible for us.

[ertsaas]

I am running into this exact same issue. I have to run Dredd tests behind a proxy and would be willing to accept the risks that this entails, so in addition to my reaction on the original post, I am adding one more reaction here 👍 !