Quarkus: "VertxException: Thread blocked" when running quarkus in ssl-only mode

[harthorst]

Describe the bug
If I disable http and enable https a lot of warnings about blocked vertex threads are shown. This happens only if I set quarkus.http.ssl-port to 0.

Expected behavior
Startup shows no warnings

Actual behavior
see attached log file

To Reproduce
Steps to reproduce the behavior:

1. git clone https://github.com/quarkusio/quarkus-quickstarts.git
2. cd quarkus-quickstarts/openapi-swaggerui-quickstart
3. change application.properties
4. run mvn quarkus:dev
   log.txt

I'm not sure if that's the right property but settingquarkus.vertx.max-worker-execute-timeto a higher value did not have any effects.

Configuration

quarkus.http.ssl-port=0
quarkus.http.insecure-requests=disabled
quarkus.http.ssl.certificate.file=localhost.crt
quarkus.http.ssl.certificate.key-file=localhost.key
# you may use server.pem and server.key file in grpc-tls-quickstart/src/main/resources/tls/ instead

Environment (please complete the following information):

• Output of uname -a or ver: Darwin xxx 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
• Output of java -version:

java version "11.0.7" 2020-04-14 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.7+8-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.7+8-LTS, mixed mode)

• Quarkus version or git rev: master
• Build tool (ie. output of mvnw --version or gradlew --version):

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /Users/thf/bin/apache-maven-3.6.3
Java version: 11.0.7, vendor: Oracle Corporation, runtime: /Users/thf/bin/jdk-11.0.7.jdk/Contents/Home
Default locale: de_DE, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.4", arch: "x86_64", family: "mac"

cc @famod

[famod]

cc @tkalmar

[famod]

@stuartwdouglas @geoand @gsmet any clue why this is happening? We also get this when starting the runner jar.
Together with #9889 we are kind of stuck because we need dynamic port(s) for a certain setup.

Btw, @harthorst did not attach the certifcate files because we used internal ones we did not want to share. Hope that's not a problem...

[stuartwdouglas]

@vietj It looks like this might be a vert.x issue. It seems that it is really slow to initialize the keystore, and this happens in a sync block so it blocking all the threads.

[famod]

FWIW: While debugging, @harthorst was seeing _multiple_ attempts to open ports (with various numbers) when ssl is active.
This could explain both this delay _and_ #9889.

He is re-checking this currently.

[stuartwdouglas]

I think this is a vert.x issue, and will need to be fixed in vert.x

[gsmet]

@vietj could you put this one on your radar?

[MastanaGuru]

I am also seeing the issue
quarkus.http.ssl-port=9443 quarkus.http.ssl.certificate.file=/var/run/secrets/openshift.io/app-certs/tls.crt quarkus.http.ssl.certificate.key-file=/var/run/secrets/openshift.io/app-certs/tls.key quarkus.http.ssl.certificate.key-store-file=/var/run/secrets/java.io/keystores/keystore.jks quarkus.http.ssl.certificate.key-store-password=${KEYSTORE_PASSWORD:${DEFAULTVALUE}} quarkus.http.insecure-requests=redirect

[stuartwdouglas]

This should fix the underlying vert.x issue https://github.com/eclipse-vertx/vert.x/pull/3494

It is actually more serious than it appears, when wildcard ports are in use each IO thread opens a new socket, so you will only have a single thread servicing the reported address.

[vietj]

@stuartwdouglas can you point out where Quarkus creates a Verticle for the HTTP server ? your PR is interesting and make some assumption about the current context which can seems ok, although I would like to investigate if there is not a better way to do.

[stuartwdouglas]

https://github.com/quarkusio/quarkus/blob/7b5b6ab9537d970c559f7d6c676f49e8a2a8a203/extensions/vertx-http/runtime/src/main/java/io/quarkus/vertx/http/runtime/VertxHttpRecorder.java#L761

[vietj]

thanks, I think the patch you provided is the most appropriate way to go @stuartwdouglas

[MastanaGuru]

@stuartwdouglas, I have two quarkus applications (a rest service and a graphql) deployed in openshift with ssl enabled

DEFAULTVALUE=mydefault
quarkus.http.ssl-port=9443
quarkus.http.ssl.certificate.file=/var/run/secrets/openshift.io/app-certs/tls.crt
quarkus.http.ssl.certificate.key-file=/var/run/secrets/openshift.io/app-certs/tls.key
quarkus.http.ssl.certificate.key-store-file=/var/run/secrets/java.io/keystores/keystore.jks
quarkus.http.ssl.certificate.key-store-password=${KEYSTORE_PASSWORD:${DEFAULTVALUE}}
quarkus.http.insecure-requests=redirect

Graphql application does not give the vertx exceptions, but the rest service does. Attached logs from both apps.

Quarkus-RestService.txt

Quarkus-graphql.txt

Anything I am doing wrong ?

[MastanaGuru]

I reinstalled everything, and dont see the exception any more.

[famod]

@stuartwdouglas Is this fixed? It seems so because vert.x was updated to 3.9.2 in Quarkus 1.7.0.CR1: #10851
See also: https://github.com/eclipse-vertx/vert.x/commits/3.9.2

[stuartwdouglas]

Yes, this should be fixed now.