Reactor-netty: Proxy settings via system properties not working with Spring WebClient

[VsKrishna87]

My following WebClient is working fine with internet connection but not through our proxy connection.

WebClient webClient = WebClient.builder()
        .baseUrl("https://targetsite.com")
        .build();

webClient.post()
    .uri("/service/serviceName")
    .body(BodyInserters.fromObject(reqData))
    .retrieve()
    .bodyToMono(WebServiceResponse.class)

Event though, the same client is working through proxy, if I set it as mentioned below,

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

ReactorClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);

WebClient webClient = WebClient.builder()
        .clientConnector(connector)
        .baseUrl("https://targetsite.com")
        .build();

webClient.post()
    .uri("/service/serviceName")
    .body(BodyInserters.fromObject(reqData))
    .retrieve()
    .bodyToMono(WebServiceResponse.class)

But if I set the same Proxy details either using System.setProperty("http.proxyHost","ourproxy.com"); System.setProperty("http.proxyPort","8080"); or JVM run-time arguments -Dhttp.proxyHost=ourproxy.com -Dhttp.proxyPort=8080

WebClient webClient = WebClient.builder()
        .baseUrl("https://targetsite.com")
        .build();

System.setProperty("http.proxyHost", "ourproxy.com");
System.setProperty("http.proxyPort", "8080");

webClient.post()
    .uri("/service/serviceName")
    .body(BodyInserters.fromObject(reqData))
    .retrieve()
    .bodyToMono(WebServiceResponse.class)

The calls are getting failed with UnknownHostException like,

[04/11/2019 12:32:43.031 IST] DEBUG [reactor-http-epoll-3] [PooledConnectionProvider:254] - Creating new client pool [http] for targetsite.com:443
[04/11/2019 12:32:43.033 IST] DEBUG [reactor-http-epoll-3] [PooledConnectionProvider:254] - [id: 0xe4a0dc15] Created new pooled channel, now 0 active connections and 1 inactive connections
[04/11/2019 12:32:43.045 IST] DEBUG [reactor-http-epoll-3] [SslProvider:254] - [id: 0xe4a0dc15] SSL enabled using engine SSLEngineImpl and SNI targetsite.com:443
[04/11/2019 12:32:43.046 IST] DEBUG [reactor-http-epoll-3] [BootstrapHandlers:254] - [id: 0xe4a0dc15] Initialized pipeline DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.sslReader = reactor.netty.tcp.SslProvider$SslReadHandler), (BootstrapHandlers$BootstrapInitializerHandler#0 = reactor.netty.channel.BootstrapHandlers$BootstrapInitializerHandler), (SimpleChannelPool$1#0 = io.netty.channel.pool.SimpleChannelPool$1), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.left.decompressor = io.netty.handler.codec.http.HttpContentDecompressor), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
[04/11/2019 12:32:43.165 IST] ERROR [reactor-http-epoll-2] [AbstractErrorWebExceptionHandler:117] - [13ebf1eb] 500 Server Error for HTTP POST "/service/serviceName"
java.net.UnknownHostException: targetsite.com: Name or service not known

Please help, why my code is not functioning even I set the proxy details via JVM run-time arguments or system properties.

Actually I want to avoid code level proxy setting. So please guide me to correct my code or approach, so that I can use JVM run-time argument option.

[violetagg]

Only this is supported (system properties are not supported)

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

You can easily add/remove proxy configuration using the system properties in your code.

[nhomble]

@violetagg just curious, do we ever see the system properties to be pulled in by reactor-netty or do you expect frameworks like spring's webclientbuilder to expose this?

[rstoyanchev]

@nhomble I would say no. It's not customary to check System properties in the Spring Framework. Typically external properties from different sources (environment, properties file, etc) are injected into classes that can then configure things accordingly.

[nhomble]

I totally agree @rstoyanchev the properties wouldn't come directly from system properties. I was thinking reactor-netty could consider using the system properties directly but integration with spring should rely on configuration properties.

I have this working in my project with the config prop approach, but I wanted to make sure I am not duplicating any enhancements either reactor or spring would consider taking in.

[violetagg]

@nhomble I prefer to have only an explicit configuration using the API instead of system properties to avoid mixture of priorities .... or should it be a choice between those two configurations or a merge of those two?

1. Also let say that we have more than one client in one JVM, then should these system properties apply to all clients or only to some of them ...

[nhomble]

Great points @violetagg. For (1) I think the client builder could support both styles and to avoid the merging/ambiguity problem the builder could either take the config as-is inTcpConfiguration with .proxy(Consumer<..>) or explicitly .proxyWithSystemProperties() for lack of a better name.

For (2), I think we can continue to be flexible by not assuming system properties are a source of configuration.

My current application just provides a proxy consumer that loads the env vars manually which I figure would belong in the library. I also imagine this would help teams like mine migrate easier 😄

[violetagg]

@nhomble

explicitly .proxyWithSystemProperties() for lack of a better name.

I like the idea and will reopen the issue. So if somebody wants can provide a PR.

[violetagg]

https://docs.oracle.com/javase/7/docs/api/java/net/doc-files/net-properties.html

[nhomble]

@violetagg thanks - do you have any thoughts around reconciling the http and https settings? The proxy enum only has http. Similar to the socket enums, should we introduce a new enum for https? Then I think it's clean to call the systemProperties method on the builder for that type.

[violetagg]

@nhomble Netty provides only one type of HttpProxyHandler and it uses CONNECT method.
https://netty.io/4.1/api/io/netty/handler/proxy/package-summary.html

[kmariappan2016]

Only this is supported (system properties are not supported)

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

You can easily add/remove proxy configuration using the system properties in your code.

I am using spring webflux latest and the reactor netty as well.But its not using Proxy

[violetagg]

@kmariappan2016 You mean that proxy functionality is not working?

[kmariappan2016]

@kmariappan2016 You mean that proxy functionality is not working?

Yes. I have configured the proxy like the following and still its not taking the proxy.
HttpClient httpClient = HttpClient.create()
.tcpConfiguration(tcpClient -> tcpClient.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)
.doOnConnected(conn -> conn.addHandlerLast(new ReadTimeoutHandler(2000, TimeUnit.MILLISECONDS)))
.proxy(proxy -> proxy.type(ProxyProvider.Proxy.HTTP).host("ourproxy").port(9000)));

    ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
    return WebClient.builder().clientConnector(connector).filter(oauth2FilterFunction).build();

Its not redirecting via proxy server.

I am using the following library
implementation 'org.springframework.security:spring-security-oauth2-client:5.3.4.RELEASE'
// https://mvnrepository.com/artifact/org.springframework/spring-webflux
implementation 'org.springframework:spring-webflux:5.2.8.RELEASE'
// https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty
implementation 'io.projectreactor.netty:reactor-netty:0.9.11.RELEASE'

I am not sure if I am missing anything
Please advice me.
Thanks

[violetagg]

@kmariappan2016 Create this as a separate issue and let's investigate there. Ok?

[kmariappan2016]

@kmariappan2016 Create this as a separate issue and let's investigate there. Ok?

Create a new issue,seems that any version above spring-security-oauth2-client:5.3.0.RELEASE is bypassing proxy server settings.

[nagyzsolthun]

tcpConfiguration is deprecated. What is the recommended approach for adding proxy config?

https://projectreactor.io/docs/netty/release/api/index.html?deprecated-list.html states:
_Apply a TcpClient mapping function to update TCP configuration and return an enriched HttpClient to use._

Can you please provider an example? Similar to

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

[violetagg]

@nagyzsolthun Everything that you was able to configure via tcpConfiguration you can configure now (since Reactor Netty version 1.0.0) directly on HttpClient level.

HttpClient httpClient =
    HttpClient.create()
              .proxy(proxy -> proxy.type(ProxyProvider.Proxy.HTTP)
                                   .host("ourproxy.com")
                                   .port(8080));

[preethirajaram]

@nagyzsolthun Everything that you was able to configure via tcpConfiguration you can configure now (since Reactor Netty version 1.0.0) directly on HttpClient level.

HttpClient httpClient =
    HttpClient.create()
              .proxy(proxy -> proxy.type(ProxyProvider.Proxy.HTTP)
                                   .host("ourproxy.com")
                                   .port(8080));

Hi,
I tried the above code with below combination,

1. reactor-netty version 1.0.0 and spring-security-test

httpClient = HttpClient.create()
.proxy(proxy -> proxy.type(ProxyProvider.Proxy.valueOf("HTTP"))
.host("abc.com")
.port(8080)

Am getting the below error:

Description:

Field context in com.db.ct.cdh.util.security.WebSecurityConfiguration$X509Endpoints required a bean of type 'javax.servlet.http.HttpServletRequest' that could not be found.

The injection point has the following annotations:
- @org.springframework.beans.factory.annotation.Autowired(required=true)

Action:

Consider defining a bean of type 'javax.servlet.http.HttpServletRequest' in your configuration.

Please suggest.

[violetagg]

Hi,
I tried the above code with below combination,

1. reactor-netty version 1.0.0 and spring-security-test

httpClient = HttpClient.create()
.proxy(proxy -> proxy.type(ProxyProvider.Proxy.valueOf("HTTP"))
.host("abc.com")
.port(8080)

Am getting the below error:

Description:

Field context in com.db.ct.cdh.util.security.WebSecurityConfiguration$X509Endpoints required a bean of type 'javax.servlet.http.HttpServletRequest' that could not be found.

The injection point has the following annotations:

• @org.springframework.beans.factory.annotation.Autowired(required=true)

Action:

Consider defining a bean of type 'javax.servlet.http.HttpServletRequest' in your configuration.

Please suggest.

@preethirajaram

Field context in com.db.ct.cdh.util.security.WebSecurityConfiguration$X509Endpoints required a bean of type 'javax.servlet.http.HttpServletRequest' that could not be found.

This is not related to Reactor Netty

[agarwaljayant7]

in my case connection seems to be not getting established between my local system/WebClient to the remote proxy server.
"[id: 0x62f7d272, L:/51.135.65.49:52907 ! R:abc-proxy.uk.xyz.abcintra.net/59.11.19.11:3128] The connection observed an error". It seems the connection b/w my local WebClient and proxy server is not getting established. Note; i've not pasted here the exact Local and Remote addresses and have mentioned the sample one. Thanks!

underlying Code snippet :

final ReactorClientHttpConnector connector = new ReactorClientHttpConnector( HttpClient.create() .tcpConfiguration( tcpClient -> tcpClient.proxy( proxy -> proxy .type(ProxyProvider.Proxy.HTTP) .host("abc-proxy.uk.xyz.abcintra.net") .port(3128) ) ) );

  String tokenResponse = WebClient.builder()
  .clientConnector(connector)
  .build()
  .post()
  .uri("https://dev.api.abc.com/oauth2/token.oauth2").headers(httpHeaders -> {
    httpHeaders.add("Authorization", authHeader);    // authHeader was generated to something like "Basic jWdb2njwUiOD3jcIfNs4dDjk3sle9oRj3sjer=" 
    httpHeaders.add("Content_Type", "application/x-www-form-urlencoded");
}).body(BodyInserters.fromValue("grant_type=client_credentials")).retrieve().bodyToMono(String.class).block();

[violetagg]

@agarwaljayant7 For issues please open a new issue, for questions please use Gitter

[agarwaljayant7]

ok, tx!

[fabiohenriquebayma]

Hi, everyone!

Thank you for all information here!

I'm working with RestTemplate right now and works fine connected in a VPN which is System Proxied. So I don't need to configure anything about the existence of the proxy to make the Spring Boot working. I was using OAuth2RestTemplate but after upgrading to Spring Boot 2.4.4 it is marked as deprecated. But see, I had to write my own CustomPasswordOAuth2AuthorizedClientProvider facing all pain to handle with private methods final classes to simple get a 'Id_token' value that I can get with two lines of code with OAuth2RestTemplate.

So, Why they deprecated a top dependency when actually makes sense to deprecated the main feature to not cut someone else legs? Why not just keep the OAuth2RestTemplate for the ones that are not importing Webflux yet cause they architecture are not async yet? I miss the concept of @Legacy instead of @Deprecated. And in my case, I don't want to be forced to pass my password to a http proxy because WebClient needs...

So, I faced issues with WebClient and Java 8 if the system has a system proxy and using Java 11 I could workaround configuring it before the application startup. But unfortunately, I had to remove 11 because some reactor ran in the test with Java 11 instead of 8 which is the version of my target source and source compatibility configuration and IntellJ configuration.

Now, I rolled back to RestTemplate until we have this problem resolved.

This code worked with Java 11