Reactor-netty: SpringBoot 2.4.2 BlockHound: blocking call when making outbound HTTPS request
Hello Reactor Netty Team,
During a test we were doing, we encountered an issue, and wanted to let you know.
This blocking call only happens when making outbound (I am the client, making a request to another server) https call.
And https is important, as it is not reproducible with http.
Here is the stack trace:
Caused by: reactor.blockhound.BlockingOperationError: Blocking call! java.io.FileInputStream#readBytes
at java.base/java.io.FileInputStream.readBytes(FileInputStream.java) ~[na:na]
at java.base/java.io.FileInputStream.read(FileInputStream.java:279) ~[na:na]
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:133) ~[na:na]
at java.base/sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424) ~[na:na]
at java.base/sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(NativePRNG.java:526) ~[na:na]
at java.base/sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:545) ~[na:na]
at java.base/sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:220) ~[na:na]
at java.base/java.security.SecureRandom.nextBytes(SecureRandom.java:741) ~[na:na]
at java.base/java.security.SecureRandom.next(SecureRandom.java:798) ~[na:na]
at java.base/java.util.Random.nextInt(Random.java:329) ~[na:na]
at java.base/sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:117) ~[na:na]
at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:297) ~[na:na]
at io.netty.handler.ssl.JdkSslClientContext.newSSLContext(JdkSslClientContext.java:294) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.JdkSslClientContext.<init>(JdkSslClientContext.java:272) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:824) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:614) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at reactor.netty.tcp.SslProvider.<init>(SslProvider.java:366) ~[reactor-netty-core-1.0.4.jar:1.0.4]
at reactor.netty.tcp.SslProvider.updateDefaultConfiguration(SslProvider.java:95) ~[reactor-netty-core-1.0.4.jar:1.0.4]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:234) ~[reactor-netty-http-1.0.4.jar:1.0.4]
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57) ~[reactor-core-3.4.3.jar:3.4.3]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.4.3.jar:3.4.3]
at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:76) ~[reactor-core-3.4.3.jar:3.4.3]
at reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46) ~[reactor-core-3.4.3.jar:3.4.3]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[reactor-core-3.4.3.jar:3.4.3]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:269) ~[reactor-netty-http-1.0.4.jar:1.0.4]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.4.3.jar:3.4.3]
May I ask what might be the root cause of this blocking call in a reactive stack for https please?
Thank you
patpatpat123
All 20 comments
Java's SSL implementation is reading random data from random device. While random device can block, it's extremely unlikely to do so (especially if you use /dev/urandom instead of /dev/random - urandom never blocks)
You could avoid this particular warning by subscribing to client outside event loop (subscribeOn instead of subscribe)
skorhone
on 16 Mar 2021
hello @skorhone,
Many thanks for your input.
Regarding your first point, "/dev/urandom instead of /dev/random" I am already currently starting my app this way:
java -Djava.security.egd=file:/dev/./urandom -jar /my-app.jar
Are you saying this is not the right -D please?
Also, I am actually not subscribing anything, I have a web app, SpringBoot app where I am the server. But in order to compute the response, I need to make a request (I become client at this point) to a third party https server, and this is where blockhouse barked.
The exact same with a third party http (no s) will not yield this issue.
This is why I raised the issue of a blocking call.
Please let me know if you need me to provide any other information!
patpatpat123
on 16 Mar 2021
Just bumping this up. And just asking, am I the only one who observes blocking call when making https call please?
patpatpat123
on 19 Mar 2021
@patpatpat123 How do you configure the WebClient?
violetagg
on 21 Mar 2021
Hello @violetagg,
I configured it that way:
With just the two files here, it is reproducible
(Everything fits into one java file + one pom file, so I didn't create a GitHub just for it, but if needed, let me know)
Thank you
package blockinghttps;
import io.netty.handler.ssl.ApplicationProtocolConfig;
import io.netty.handler.ssl.ApplicationProtocolNames;
import io.netty.handler.ssl.SslContextBuilder;
import nl.altindag.ssl.SSLFactory;
import nl.altindag.ssl.util.NettySslUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Primary;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.reactive.ClientHttpConnector;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.blockhound.BlockHound;
import reactor.core.publisher.Mono;
import reactor.netty.http.HttpProtocol;
import reactor.netty.http.client.HttpClient;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.function.Function;
@RestController
@SpringBootApplication
public class MyApplication {
@Autowired
WebClient webClient;
static {
BlockHound.install();
}
/**
* The entry point of application.
* Second micro service to deploy
*
* @param args the input arguments
*/
public static void main(String[] args) {
SpringApplication.run(MyApplication.class);
}
@PostMapping(path = "/blockinghttps", produces = MediaType.APPLICATION_JSON_VALUE)
public Mono<ResponseEntity<String>> refundCustomer() {
Mono<String> s = webClient.mutate().baseUrl("https://google.com").build().post().uri("/route").retrieve().bodyToMono(String.class);
return s.map(one -> ResponseEntity.ok(one));
}
@Bean
@Primary
public WebClient getWebClient(ClientHttpConnector clientHttpConnector) {
return WebClient.create().mutate().defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE, HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE).clientConnector(clientHttpConnector).build();
}
@Bean
public SSLFactory getInternalSslFactory() {
try (InputStream newInputStreamKeyStorePathToInternal = Files.newInputStream(Paths.get(URI.create("file:///Users/abc/keystore.p12"))); InputStream newInputStreamTrustStorePath = Files.newInputStream(Paths.get(URI.create("file:///Users/abc/truststore.p12")))) {
return SSLFactory.builder().withIdentityMaterial(newInputStreamKeyStorePathToInternal, "".toCharArray(), "".toCharArray()).withTrustMaterial(newInputStreamTrustStorePath, "12345678".toCharArray()).build();
} catch (IOException | IllegalArgumentException e) {
return null;
}
}
@Bean
public SslContextBuilder getSslContextBuilder(SSLFactory sslFactory) {
final ApplicationProtocolConfig applicationProtocolConfig = new ApplicationProtocolConfig(ApplicationProtocolConfig.Protocol.ALPN, ApplicationProtocolConfig.SelectorFailureBehavior.NO_ADVERTISE, ApplicationProtocolConfig.SelectedListenerFailureBehavior.ACCEPT, ApplicationProtocolNames.HTTP_2, ApplicationProtocolNames.HTTP_1_1);
return NettySslUtils.forClient(sslFactory).applicationProtocolConfig(applicationProtocolConfig);
}
@Bean
public HttpClient getHttpClient(SslContextBuilder sslContextBuilder) {
return HttpClient.create().wiretap(false).metrics(true, Function.identity()).protocol(HttpProtocol.HTTP11).secure(sslContextSpec -> sslContextSpec.sslContext(sslContextBuilder));
}
@Bean
@Primary
public ClientHttpConnector getClientHttpConnector(HttpClient httpClient) {
return new ReactorClientHttpConnector(httpClient);
}
}
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>blockinghttps</artifactId>
<version>1.0-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.0-M3</version>
<relativePath/>
</parent>
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>io.github.hakky54</groupId>
<artifactId>sslcontext-kickstart-for-netty</artifactId>
<version>6.4.0</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
<version>1.6.5</version>
</dependency>
<dependency>
<groupId>io.projectreactor.tools</groupId>
<artifactId>blockhound</artifactId>
<version>1.0.4.RELEASE</version>
</dependency>
</dependencies>
<repositories>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>https://repo.spring.io/snapshot</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>https://repo.spring.io/milestone</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>https://repo.spring.io/snapshot</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
<pluginRepository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>https://repo.spring.io/milestone</url>
</pluginRepository>
</pluginRepositories>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
@patpatpat123 Can you provide SslContext instead of SslContextBuilder? I see that you configure everything for HTTP/2 (although I do not see the ciphers).
violetagg
on 22 Mar 2021
Thank you for taking a look at this @violetagg, much appreciated.
Indeed, I seems if I go from SslContextBuilder to use SslContext, the blocking call disappear.
We are actually using SslContextBuilder because of your recommendation here:
https://github.com/reactor/reactor-netty/issues/1500
We are using http2. Hence, I was wondering: is it possible to not block with SslContextBuilder as well?
patpatpat123
on 22 Mar 2021
Indeed, I seems if I go from SslContextBuilder to use SslContext, the blocking call disappear.
We are actually using SslContextBuilder because of your recommendation here:
1500
@patpatpat123 Yep I remember that
But you already have almost everything from that default configuration ... See my comment here.
https://github.com/reactor/reactor-netty/issues/1500#issuecomment-776538529
We are using http2. Hence, I was wondering: is it possible to not block with SslContextBuilder as well?
Can you try with subscribeOn?
webClient.mutate()
.baseUrl("https://google.com")
.build()
.post()
.uri("/route")
.retrieve()
.bodyToMono(String.class)
.subscribeOn(Schedulers.boundedElastic())
I got the same issue but cannot use that workaround since I'm using webflux with kotlin + coroutines:
Stacktrace:
Caused by: reactor.blockhound.BlockingOperationError: Blocking call! java.io.FileInputStream#readBytes
at java.base/java.io.FileInputStream.readBytes(FileInputStream.java) ~[na:na]
at java.base/java.io.FileInputStream.read(FileInputStream.java:279) ~[na:na]
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[na:na]
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271) ~[na:na]
at java.base/sun.security.util.DerValue.init(DerValue.java:388) ~[na:na]
at java.base/sun.security.util.DerValue.<init>(DerValue.java:331) ~[na:na]
at java.base/sun.security.util.DerValue.<init>(DerValue.java:344) ~[na:na]
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1993) ~[na:na]
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222) ~[na:na]
at java.base/java.security.KeyStore.load(KeyStore.java:1479) ~[na:na]
at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365) ~[na:na]
at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313) ~[na:na]
at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55) ~[na:na]
at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[na:na]
at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:278) ~[na:na]
at java.base/sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:88) ~[na:na]
at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:297) ~[na:na]
at io.netty.handler.ssl.JdkSslContext.<clinit>(JdkSslContext.java:75) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:824) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:614) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
at reactor.netty.tcp.SslProvider.<init>(SslProvider.java:299) ~[reactor-netty-core-1.0.5.jar:1.0.5]
at reactor.netty.tcp.SslProvider$Build.build(SslProvider.java:668) ~[reactor-netty-core-1.0.5.jar:1.0.5]
at reactor.netty.http.client.HttpClientSecure.<clinit>(HttpClientSecure.java:88) ~[reactor-netty-http-1.0.5.jar:1.0.5]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:219) ~[reactor-netty-http-1.0.5.jar:1.0.5]
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:76) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:271) ~[reactor-netty-http-1.0.5.jar:1.0.5]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.4.4.jar:3.4.4]
at reactor.core.publisher.Mono.subscribe(Mono.java:4099) ~[reactor-core-3.4.4.jar:3.4.4]
at kotlinx.coroutines.reactive.AwaitKt.awaitOne(Await.kt:137) ~[kotlinx-coroutines-reactive-1.4.3.jar:na]
at kotlinx.coroutines.reactive.AwaitKt.awaitOne$default(Await.kt:135) ~[kotlinx-coroutines-reactive-1.4.3.jar:na]
at kotlinx.coroutines.reactive.AwaitKt.awaitSingle(Await.kt:81) ~[kotlinx-coroutines-reactive-1.4.3.jar:na]
at org.springframework.web.reactive.function.client.WebClientExtensionsKt.awaitExchange(WebClientExtensions.kt:91) ~[spring-webflux-5.3.5.jar:5.3.5]
My configuration is for the client is the default, I'm simply injecting the auto configurable webclient and making a get to a https url:
webClient.get().uri("/1234").awaitExchange { it.statusCode() }
Since I'm using the reactor kotlin extension instead of Mono/Flux I have no easy way to change like you suggested.
Will this blocking call be a real issue? Can you please advise any workaround other than changing to subscribeOn? I'm not also comfortable of creating a custom bean for the SslContext, I'd prefer using the default implementation
Edit: I could launch a coroutine in another context (e.g Dispatchers.IO), but wanted to understand first if this is would be a real danger
rp199
on 23 Mar 2021
To update this thread, yes @violetagg, you are correct, with your subscribe on solution, it seems to work.
But it seems to be more of a workaround.
Same with @rp199, would it be possible to have some sort of real fix, and not a workaround instead?
Just asking. And thank you again for the help provided.
patpatpat123
on 23 Mar 2021
@rp199 Do you use some SSL configuration or you use the defaults?
violetagg
on 24 Mar 2021
@violetagg I'm using the web client without any customisation, using the builder directly from the WebClientAutoConfiguration:
@Bean
fun webClient(
wWebClient: WebClient.Builder
) = webClient.baseUrl("https://some-endpoint.com").build()
Don't have any custom configuration related to SSL.
rp199
on 24 Mar 2021
@rp199 In your use case, you expect the HttpClient to decide lazy based on the scheme whether you need security or not.
Can you try the snippet below (it is in java, but the idea is important)
WebClient.builder().clientConnector(new ReactorClientHttpConnector(HttpClient.create().compress(true).secure())).build();
Thanks @violetagg, I can confirm that snippet works, I'm no longer getting any exception from blockhound and the https call is working fine 馃檪
Just for sanity check, what was happening before was that if nothing is specified, the HttpClient will try to evaluate if we need security for the call and that's where the blocking call was happening (that's why it worked on http calls and not https). With that snippet, we enforced the HttpClient to use security.
But the end result is the same, ultimately the settings that we set would be the ones it would use if it decided for itself, right? If understand correctly, is just like a "shortcut" for the same result.
Thanks again for the help.
rp199
on 24 Mar 2021
Many thanks all.
Shall I just close this issue, and let Reactor Netty Team decide if making the SslContextBuilder non blocking can be considered for the future?
patpatpat123
on 24 Mar 2021
@patpatpat123 Let's keep this opened for the moment.
violetagg
on 24 Mar 2021
@rp199 with .secure() you eagerly initialise the default SslContext, while without it the default SslContext will be resolved lazy when there is a url with https scheme.
violetagg
on 24 Mar 2021
Understood, thanks again
rp199
on 24 Mar 2021
@patpatpat123 With PR #1573, we exposed a new API that you can use in order to utilise the default configuration and also to solve the blocking call issue:
/**
* SslContext builder that provides, specific for the protocol, default configuration
* e.g. {@link DefaultSslContextSpec}, {@link TcpSslContextSpec} etc.
* As opposed to {@link #sslContext(SslContextBuilder)}, the default configuration is applied before
* any other custom configuration.
*
* @param spec SslContext builder that provides, specific for the protocol, default configuration
* @return {@literal this}
* @since 1.0.6
*/
reactor.netty.tcp.SslProvider.SslContextSpec#sslContext(reactor.netty.tcp.SslProvider.ProtocolSslContextSpec)
Examples
HTTP/1.1
Http11SslContextSpec http11SslContextSpec =
Http11SslContextSpec.forServer(cert, key)
.configure(sslContextBuilder -> sslContextBuilder...);
HttpServer.create()
.secure(spec -> spec.sslContext(http11SslContextSpec))
HTTP/2
Http2SslContextSpec http11SslContextSpec =
Http2SslContextSpec.forServer(cert, key)
.configure(sslContextBuilder -> sslContextBuilder...);
HttpServer.create()
.secure(spec -> spec.sslContext(http11SslContextSpec))
Many thanks @violetagg. Good day!
patpatpat123
on 9 Apr 2021
Related issues
violetagg
路
8Comments
violetagg
路
4Comments
sdeleuze
路
7Comments
be-hase
路
6Comments
eugene-sadovsky
路
8Comments
Most helpful comment
@patpatpat123 With PR #1573, we exposed a new API that you can use in order to utilise the default configuration and also to solve the blocking call issue:
Examples
HTTP/1.1
HTTP/2