Wiremock: SSL exceptions when proxying requests

Created on 27 May 2020 · 15Comments · Source: tomakehurst/wiremock

Bug reports

We're a lot more likely to look at and fix bugs that are clearly described and reproduceable. Please including the following details when reporting a bug:

[x] Which version of WireMock you're using: 2.26.3
[x] How you're starting and configuring WireMock, including configuration or CLI the command line: docker-compose to run wiremock and Dockerfile to build wiremock image are provided in github repo
[x] A failing test case that demonstrates the problem: there are two curl commands in the readme file. Please run it one by one couple times.

Hello,
First of all, thank you all for the awesome software!
I have an issue when using wiremock together with haproxy which is responsible for proxying all requests (not mocked in wiremock ones).
I created a small simple docker compose file for you to test it: https://github.com/AnCh7/wiremock-haproxy
It consist of:
1) Wiremock Dockerfile with healthcheck and proxy-all mappings.
2) Haproxy config file (pretty simple configuration)
3) Docker compose file and readme file with information how to run it and error that I am getting.
Usually, if you run same curl command twice, you can get this error first time, but never second. So please run these to curl commands one by one.

Bug

Source

AnCh7

Most helpful comment

OK, I can replicate this now if I run your script. Having tinkered with it a little it seems that making requests with different Host values in short succession triggers the issue, so there may be something in the hypothesis that this is something WireMock related.

I'm going to keep looking, anyway,

tomakehurst on 10 Jun 2020

🎉2

All 15 comments

Please can you describe the actual problem you're seeing and how to reproduce it?

tomakehurst on 27 May 2020

Sure.
I am getting OpenSSL error when wiremock is proxying requests through to other hosts, for example:

haproxy_1          | 00000016:fe.clireq[0012:ffffffff]: GET /v1/public/coin/1 HTTP/1.1
haproxy_1          | 00000016:fe.clihdr[0012:ffffffff]: accept: */*
haproxy_1          | 00000016:fe.clihdr[0012:ffffffff]: user-agent: curl/7.64.1
haproxy_1          | 00000016:fe.clihdr[0012:ffffffff]: host: api.coinranking.com
haproxy_1          | fd[0014] OpenSSL error[0x1408f10b] ssl3_get_record: wrong version number
haproxy_1          | fd[0014] OpenSSL error[0x140e0197] SSL_shutdown: shutdown while in init
haproxy_1          | 00000016:be.srvcls[0012:0014]
haproxy_1          | 00000016:be.clicls[0012:0014]
haproxy_1          | 00000016:be.closed[0012:0014]
haproxy_1          | <134>May 27 15:57:53 haproxy[7]: ::ffff:172.18.0.2:54708 [27/May/2020:15:57:53.587] fe be/clear 89/0/0/-1/268 -1 0 - - SD-- 1/1/0/0/0 0/0 {13.226.207.20} "GET /v1/public/coin/1 HTTP/1.1"

To able to reproduce it please do this:
1) Clone this repo that I created:

git clone [email protected]:AnCh7/wiremock-haproxy.git

2) Go to wiremock-haproxy directory and run multi-container Docker applications by using docker-compose:

cd wiremock-haproxy
docker-compose up --build

3) Open second terminal window and run these curl commands:

curl --verbose --header 'Host: api.oceandrivers.com' 'http://localhost:9000/v1.0/getEasyWind/EW013/?period=latestdata'
curl --verbose --header 'Host: api.coinranking.com' 'http://localhost:9000/v1/public/coin/1'

4) Second curl fails with exception:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9000 (#0)
> GET /v1/public/coin/1 HTTP/1.1
> Host: api.coinranking.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 500 Server Error
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 10188
< Connection: close
< Server: Jetty(9.4.20.v20190813)
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /v1/public/coin/1. Reason:
<pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.RuntimeException: wiremock.org.apache.http.NoHttpResponseException: The target server failed to respond
    at com.github.tomakehurst.wiremock.http.ProxyResponseRenderer.render(ProxyResponseRenderer.java:94)
    at com.github.tomakehurst.wiremock.http.StubResponseRenderer.buildResponse(StubResponseRenderer.java:62)
    at com.github.tomakehurst.wiremock.http.StubResponseRenderer.render(StubResponseRenderer.java:56)
    at com.github.tomakehurst.wiremock.http.AbstractRequestHandler.handle(AbstractRequestHandler.java:72)

and log from haproxy:

haproxy_1          | 0000000d:fe.clireq[0012:ffffffff]: GET /v1/public/coin/1 HTTP/1.1
haproxy_1          | 0000000d:fe.clihdr[0012:ffffffff]: accept: */*
haproxy_1          | 0000000d:fe.clihdr[0012:ffffffff]: user-agent: curl/7.64.1
haproxy_1          | 0000000d:fe.clihdr[0012:ffffffff]: host: api.coinranking.com
haproxy_1          | 0000000d:be.srvcls[0012:0014]
haproxy_1          | 0000000d:be.clicls[0012:0014]
haproxy_1          | 0000000d:be.closed[0012:0014]
haproxy_1          | <134>May 28 18:27:23 haproxy[7]: ::ffff:172.18.0.2:43704 [28/May/2020:18:27:23.263] fe be/clear 110/0/0/-1/297 -1 0 - - SD-- 1/1/0/0/0 0/0 {13.249.87.97} "GET /v1/public/coin/1 HTTP/1.1"
haproxy_1          | fd[0014] OpenSSL error[0x1408f10b] ssl3_get_record: wrong version number
haproxy_1          | fd[0014] OpenSSL error[0x140e0197] SSL_shutdown: shutdown while in init

5) Run same last curl command again:

curl --verbose --header 'Host: api.coinranking.com' 'http://localhost:9000/v1/public/coin/1'

No exception this time.

AnCh7 on 28 May 2020

I can't replicate this. When I run it, WireMock proxies through to HAProxy OK and gets a 503 (no service) response.

tomakehurst on 1 Jun 2020

Hi, Tom

Can you please try it again?
Because I can still reproduce it. I also tried it on my different machines: Windows PC and Macbook with MacOs. Couple of my colleagues tried it too and got same OpenSSL issue.
Here I recorded a small screencast for you: https://github.com/AnCh7/wiremock-haproxy/blob/master/iterm.gif

AnCh7 on 5 Jun 2020

I still can't reproduce it. I ran exactly the same commands as in your screencast and it all worked fine.

tomakehurst on 5 Jun 2020

I don't think this has anything to do with WireMock TBH. Seems like an issue between HAProxy and the sites you're proxying to.

Have you tried clearing out your docker cache? Wondering if you've got an old/corrupted layer lying around for the HAProxy image.

tomakehurst on 5 Jun 2020

Hi, Tom

I added a test script to my repo, which is sending requests to couple public APIs and with it I can constantly reproduce the issue. Also added logs (logs folder in the repo) from Wiremock and HAProxy.

Docker cache - I always run docker system prune before any tests. Tried docker system prune --all as well. Same issue with OpenSSL.

HAProxy configuration - I tried different settings like tune.ssl, hold for different types, maxconn etc. No luck, same errors.

I was also exploring other tools similar to Wiremock, and tried replacing it with MockServer which also has proxying feature and I was not able to reproduce issue with it. That is why I opened this issue because I thought that it is related to Wiremock and not HAProxy.

AnCh7 on 8 Jun 2020

I'm going to keep looking, anyway,

tomakehurst on 10 Jun 2020

🎉2

Hi @tomakehurst! Just wanted to check-in if you have any other thoughts on this issue? Is there anything that we can do to help triage the issue and provide more detail? Are there specific places in the source where you think this issue might be caused so we could debug and provide more information?

Thanks again!

jgornick on 17 Jun 2020

Nope, I'd have posted on here if I did.

If you want to do your own debugging, then I'd suggest you start by looking in ProxyResponseRenderer, as this is where the forwarded requests get made.

tomakehurst on 17 Jun 2020

👍1

OK, I've found the problem.

When preserveHostHeader is set, WireMock copies the original host header to a header on the proxy request, but this isn't sufficient on its own for the Apache HTTP client to treat it as a distinct route from a connection pooling perspective. So it's attempting to reuse the connection from a call to one domain for another.

You can work around this for now by disabling connection reuse via a system property in the WireMock startup command: -Dhttp.keepAlive=false.

Edit: previous version of this comment contained a patch that turned out not to be correct.

tomakehurst on 22 Jun 2020

👍1

Good news. I am going to try it this week and I will let u know how it works. Thank you!

AnCh7 on 22 Jun 2020

👍1

-Dhttp.keepAlive=false definitely works. Since I am running wiremock as standalone app inside container, it was easy for me to add this flag and I am totally good with this workaround. Thank you again Tom.

anton-chuhunou on 26 Jun 2020

@tomakehurst Just to keep the backlog clean, should the title of this issue be renamed to something that lines up more with the preserving of host headers causes connection pooling issues?

Thanks again for helping with this!

jgornick on 15 Jul 2020

Connection reuse by the proxy client has now been completely disabled in 55490cec33dee35c031a53e13b94fd7a9c37fa16

tomakehurst on 6 May 2021

🎉1

Was this page helpful?

0 / 5 - 0 ratings