Reactor-netty: HTTP Client throws `AnnotatedConnectException`s after 1.0.1 upgrade

[hisener]

We started seeing io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.1.1:54117 on Travis CI while upgrading Reactor BOM Dysprosium-SR14 -> 2020.0.1 (reactor-netty 0.9.14 -> 1.0.1). Both reactor-netty versions pull Netty 4.1.53.Final version so I assume it's related reactor-netty.

It happens when there are multiple IP addresses to a single hostname (localhost in this case). For example

# /etc/hosts
127.0.1.1   localhost
127.0.0.1   localhost

Expected Behavior

It should be able to connect to the webserver.

Actual Behavior

Connection refused error.

Steps to Reproduce

cd /tmp
git clone [email protected]:hisener/playground.git
cd playground
docker run -it -v $(PWD)/:/opt/ -v ~/.m2/:/root/.m2/ maven:3.6.3-jdk-11-slim /bin/bash

Add 127.0.1.1 localhost to /etc/hosts. It should look like this

$ cat /etc/hosts
127.0.1.1   localhost
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2  30591ed76759

cd /opt
mvn clean package
[ERROR] Tests run: 30, Failures: 20, Errors: 0, Skipped: 0
````

The [test](https://github.com/hisener/playground/blob/master/spring-webclient-connection-failure-test/src/test/java/com/github/hisener/spring/webclient/WebClientTest.java) that fails:

```java
private final MockWebServer server = new MockWebServer();
private final URI baseUrl = server.url("").uri();

@Test
void reactorNetty() {
    server.enqueue(new MockResponse());

    reactor.netty.http.client.HttpClient.create()
            .baseUrl(baseUrl.toString())
            .get()
            .response()
            .as(StepVerifier::create)
            .assertNext(res -> assertThat(res.status()).isEqualTo(HttpResponseStatus.OK))
            .verifyComplete();
}

Possible Solution

Your Environment

• Reactor version(s) used: Reactor BOM 2020.0.1 (reactor-core:3.4.0, reactor-netty:1.0.1)
• Other relevant libraries versions (eg. netty, ...): Netty 4.1.53.Final
• JVM version (java -version): 11.0.6 + 11.0.2 (Travis CI)
• OS and version (eg. uname -a): Debian GNU/Linux 10 (buster) 4.19.76-linuxkit + Ubuntu 18.04 and 20.04 (Travis CI).

Additional Context

We realized this regression on Travis; see https://travis-ci.com/github/hisener/playground/builds/205882662 for minimal reproducible example repo tests. It passes on Ubuntu 16.04.6 and fails on 18.04.4 and 20.04.1.

# Ubuntu 16.04.6
$ cat /etc/hosts
## Managed by Chef on packer-5ef350f5-b1b4-97f2-7591-153f7d417944.c.eco-emissary-99515.internal :heart_eyes_cat:
## cookbook:: travis_build_environment
##     file:: templates/default/etc/cloud/templates/hosts.tmpl.erb

127.0.1.1 travis-job-8f8bce8d-0398-4288-a326-456dd21ce364 travis-job-8f8bce8d-0398-4288-a326-456dd21ce364 ip4-loopback xenial64

127.0.0.1 localhost nettuno travis vagrant travis-job-8f8bce8d-0398-4288-a326-456dd21ce364 

# Ubuntu 18.04.4
$ cat /etc/hosts
## Managed by Chef on packer-5efb3610-907d-b1cc-c3e4-6423ffe2ee52.c.eco-emissary-99515.internal :heart_eyes_cat:
## cookbook:: travis_build_environment
##     file:: templates/default/etc/cloud/templates/hosts.tmpl.erb

127.0.1.1 localhost localhost ip4-loopback 

127.0.0.1 localhost nettuno travis vagrant travis-job-e33bb155-60da-4ac6-ba59-fe7469df01df

# Ubuntu 20.04.1
$ cat /etc/hosts
## Managed by Chef on packer-5fc61912-5584-3eaa-aa0c-2bbfc77c799e.c.eco-emissary-99515.internal :heart_eyes_cat:
## cookbook:: travis_build_environment
##     file:: templates/default/etc/cloud/templates/hosts.tmpl.erb

127.0.1.1 localhost localhost ip4-loopback 

127.0.0.1 localhost nettuno travis vagrant travis-job-d0e2e7ee-c2cf-492f-ad64-d3d1148bf11d

There is a question about this on the Travis community page, but unfortunately, there is no response.

I have tried with Reactor BOM Dysprosium-SR14 and 2020.0.1 as well as Apache HTTP client. Here is the summary for tests for two /etc/hosts files:

127.0.0.1       localhost
127.0.1.1       localhost

• Apache 🟢
• Reactor Netty 0.9.14.RELEASE (Dysprosium-SR14) 🟢
• Reactor Netty 1.0.1 (2020.0.1) 🟢

---

127.0.1.1       localhost
127.0.0.1       localhost

• Apache 🟢
• Reactor Netty 0.9.14.RELEASE (Dysprosium-SR14) 🟢
• Reactor Netty 1.0.1 (2020.0.1) 🔴

[violetagg]

@hisener As a workaround please use HttpClient.resolver(DefaultAddressResolverGroup.INSTANCE)
I created issue in Netty https://github.com/netty/netty/issues/10834

[violetagg]

@@hisener I'm closing this one, please follow the Netty's issue.

[violetagg]

@hisener This issue should be OK now with Reactor Netty 1.0.7 and Netty 4.1.64.Final

[hisener]

Thanks for the fix in Netty! 🙏 I'll give it a shot once Netty BOM is available on Maven Central and drop workarounds.