Hono: SSL Performance of MQTT Adapter

[Christian-Schmid]

Hi Hono Devs,

I recently did some load testing with the mqtt adapter and got some interesting results.

The test setup looks as follows:

• 10k simulated devices which produce a total load of 2000 messages (512 bytes each) per second.
• 2 MQTT adapter instances are deployed
• The test runs for 10 minutes

Test Run 1: No SSL enabled:

• The MQTT Adapters can handle the load.

Test Run 2: SSL enabled:

• The MQTT Adapters are not able to handle the load:
  
• They are able to handle about 200 messages (100 per adapter) per second

Test Run 3 SSL based on Netty BoringSSL Binding

• After reading the following issue: HTTP+SSL performance issue on the Vert.X Mailing list I patched the hono mqtt adapter to also use the boring ssl netty binding (see http://netty.io/wiki/forked-tomcat-native.html ) and got the following numbers:

• The mqtt adapters were able to handle the scenario again.

I just tested with the MQTT Adapter, but i guess it should also affect the REST adapter

My question now:

• Do you have experience / Ideas how to tweak the java implementation for better performance?
• Alternatively: What would you think about a configuration option to allow the user to configure the used ssl implementation.

Thanks Chris

[sophokles73]

This sounds like a great idea. We can use the OpenSSL engine in the Docker images we create by default, but we will still need to make it configurable in the adapters' config properties.

Would you care to create a PR?

[Christian-Schmid]

Hi,

thanks for your fast response, yes I can prepare a pr.

There are different maven dependencies for the openssl utilization available on netty: http://netty.io/wiki/forked-tomcat-native.html

I would go for the "netty-tcnative-{os_arch}" (default) variant. It requires OpenSSL to be installed on the underlying OS layer and it can also be updated / patched there.

• The BoringSSL variant seems not to be intended for usage in 3rd party applications, so I guess patch / security management would be difficult.

• The OpenSSL static variant requires us to rebuild the hono adapter in case of OpenSSL patches.

• The LibreSSL variant doesn't seem to be finished :-)

Therefore I would go for variant 1 with the dynamic open SSL binding

[sophokles73]

I agree in general. My understanding is that this would require the underlying OS docker image to include a working OpenSSL installation, right? Not sure if the (currently) used image (openjdk:8u162-jre-slim) supports it ...

[sophokles73]

It seems like the image is lacking libapr1. We could, however, install it on the base image as part of our build process ...

[Christian-Schmid]

Yes the used docker image would need a working OpenSSL and libapr1 installation.

Are you proposing to create a hono specific base image based on {{openjdk:*-jre-slim}} with the required library(ies) installed, or should we try to add the library during the builds of the adapters dynamically (e.g. with the fabric8 plugin) ?
If the set up is not complex I would go for the moment with the second option

[sophokles73]

If the set up is not complex I would go for the moment with the second option

So do I :+1:

[Christian-Schmid]

I tried my luck in using the provided netty-tcnative artifact.

Sadly this doesn't seem to integrate very well with the base image we're using. The hono base image openjdk:8u162-jre-slim is already using openssl 1.1.* while the dependency still relies on the openssl 1.0.* branch.

Furthermore I didn't manage to get it working with a downgraded version. Maybe this is because of some misaligned dependencies (vert.x / netty).

For the moment I would therefore use the boringssl-static dependency. (The OpenSSL static dependency is not available on maven central, and we would have to build it our self).
Some other projects are using BoringSSL too ( https://mvnrepository.com/artifact/io.netty/netty-tcnative-boringssl-static/2.0.7.Final/usages )

WDYT?

[sophokles73]

what I do not like particularly much about this idea is that it would introduce several pre-req dependencies to Hono which we need to get through the IP process at Eclipse.

[sophokles73]

Quote from the boringssl page:

BoringSSL is a fork of OpenSSL that is designed to meet Google's needs.

Although BoringSSL is an open source project, it is not intended for general use, as OpenSSL is. We don't recommend that third parties depend upon it. Doing so is likely to be frustrating because there are no guarantees of API or ABI stability.

[Christian-Schmid]

I'm not too familiar with the Eclipse IP process, but Vert.x is also using the dependency. Maybe they already have done the necessary paperwork ;-)

I stumbled also across the project description of Boring SSL, which was the main reason why I wanted to use the OpenSSL native version in the first place. But we're not directly relying on the ABI nor the API. So I think the (hard) integration work is done by the maintainers of the netty-tcnative project.

I think, best would be to ask on tcnative mailing list what their current recommendation is.

[sophokles73]

I'm not too familiar with the Eclipse IP process, but Vert.x is also using the dependency. Maybe they already have done the necessary paperwork ;-)

They only use it for running some test cases which means that they didn't need to take it through the whole process.

I am currently experimenting with a fedora based openjdk image (oracle/openjdk:8) which provides teh right version of libssl. However, I am not (yet) able to get it running ...

[sophokles73]

Here's an update: I haven't been able to get the dynamically linked version of netty-tcnative running in any of the combinations I have tried. What does work consistently though, is using a statically linked version.
So, the question now remaining is, if we can prevent requiring Hono to include the statically linked artifact (either openssl or boringssl) in our Docker images. The reason I would like to prevent this is because we would then need to get the source code of the linked libraries through the Eclipse IP process, which I would rather not like to do.

@Christian-Schmid maybe you can experiment a little with options for including the required artifact in a custom Docker image built on top of the standard Hono image which would include the jar in Hono's classpath.

[ctron]

I am currently experimenting with a fedora based openjdk image (oracle/openjdk:8) which provides teh right version of libssl. However, I am not (yet) able to get it running ...

Can you share what the problem is?

[sophokles73]

@ctron the problem is that during startup of e.g. the Auth server component, netty complains that KeyManagerFactory is not supported and thus the server's key and cert cannot be set. I have googled a whole day for this problem but it basically seems to be related to the fact that there is no such thing as a particular version of libssl. Every distro seems to roll their own so finding a distro that works with the dynamically linked version of tcnative seems to be a game of pure luck ...

[ctron]

But be missing KeyManagerFactory that sounds more like a second level problem. I would assume there is a different root cause which prevents the class loader from loading the class.

Do you have a simple reproducer?

[sophokles73]

This is not about not being able to load the KeyManagerFactory class. The problem is related to netty probing the underlying libssl for certain features. In this case, it fails to detect support for using a KeyManagerFactory for setting the server's key and cert.

You can use the use_openssl branch and run the SslTest unit test contained in the Auth Server module. Its POM file declares two alternative dependencies: one for the statically linked BoringSSL based tcnative lib (which supports KeyManagerFactory and works for running the service) and one for the dynamically linked OpenSSL version (which on my machine finds OpenSSL 1.0.2n but does not support KeyManagerFactory and thus does not work for running the service). Your mileage on your machine may vary depending on which version of OpenSSL you have installed (or what version your distro tries to make you believe you have for that matter ...). FMPOV the only option is to include the BoringSSL based version in Hono in order to be independent from the (seemingly totally broken) OpenSSL ecosystem ...

[Christian-Schmid]

I tried my luck also with different types of base images (oracle/openjdk:8, openjdk:8u162-jre-slim, even with a basic ubuntu, fedora) and stumbled across the same problems you experienced. I guess we would have to use a base image which has a maintained tcnative package in their repository. For this then we would have to find a matching maven dependency for netty.

I just created the pr https://github.com/eclipse/hono/pull/639 which allows configuration of native tls (without any concrete implementation added to the dependencies). Maybe this is also helpful for @ctron to experiment with it, it shows how to enable the native tls.

@sophokles73: I can have another short look at the docker images. Maybe there is already one for netty out here, and we can have a look how they solved it there.

[sophokles73]

I can have another short look at the docker images. Maybe there is already one for netty out here, and we can have a look how they solved it there.

You mean a Docker image that comes with a pre-configured, working BoringSSL or OpenSSL based tcnative and Java 8? That would be nice ...

[ctron]

Thanks, that SslTest helped to understand the issue. This is indeed a linking issue:

Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_tcnative8413922882652429363.so: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Which is caused by:

~
linux-vdso.so.1 => (0x00007ffea0fdb000)
libssl.so.1.0.0 => not found
libcrypto.so.1.0.0 => not found
libapr-1.so.0 => /lib64/libapr-1.so.0 (0x00007fee9b91c000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fee9b700000)
libc.so.6 => /lib64/libc.so.6 (0x00007fee9b333000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fee9b12e000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fee9aef7000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fee9acf3000)
/lib64/ld-linux-x86-64.so.2 (0x00007fee9bd65000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007fee9aaf0000)
~

I would assume that this is cause on other distributions as well.

However I am not convinced that simply dropping in another, statically linked, version of a crypto library is the right approach to this issue. Security issues fixed by the distribution would no longer be effective and a Hono update would be require instead.

From what I've seen there are a few alternatives in the netty-tcnative project.

[ctron]

What works for me us using:

~xml

io.netty
netty-tcnative
linux-x86_64-fedora
2.0.8.Final

~

And I am not using Fedora. So that might just be bad naming.

[ctron]

When I do a local build of the "openssl-dynamic" project I get a working version even without the "fedora" classifier:

~
OpenSSL 1.0.2k-fips 26 Jan 2017
supports hostname validation: true
supports KeyManagerFactory: true
~

I would say that this is the way to go.

[sophokles73]

I would assume that this is cause on other distributions as well.

No, the problem is not that the library is not found. The problem is that the library does not support the required features. When I run the test using the dynamically linked version on my local Ubuntu 18.04 laptop, I get

OpenSSL 1.0.2n  7 Dec 2017
supports hostname validation: false
supports KeyManagerFactory: false

Interestingly, when I use the latest netty-tcnative-openssl-static from the project's CI server, I get

OpenSSL 1.0.2n  7 Dec 2017
supports hostname validation: true
supports KeyManagerFactory: true

which for me illustrates the whole silliness of openSSL.

Security issues fixed by the distribution would no longer be effective and a Hono update would be require instead.

We are not supporting this use case anyway because we are shipping Docker images. The way to update the OpenSSL lib used is to re-build the Docker image, regardless of whether the OpenSSL lib is maintained as part of the OS image or as part of the layer added by Hono, isn't it?

[ctron]

Can you try replacing the base image with ctron/java-base-tcnative:latest, which is built by:

~~~Dockerfile
FROM openjdk:jdk-slim

MAINTAINER "Jens Reimann [e-mail]"

RUN apt-get -y update
RUN apt-get -y dist-upgrade
RUN apt-get -y install wget git

RUN wget https://www-eu.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz
RUN tar xzf apache-maven-3.5.3-bin.tar.gz

RUN git clone https://github.com/netty/netty-tcnative
WORKDIR netty-tcnative

RUN git checkout netty-tcnative-parent-2.0.8.Final
RUN apt-get -y install autoconf automake libtool libapr1-dev libssl-dev make
RUN /apache-maven-3.5.3/bin/mvn -B clean install -f openssl-dynamic/pom.xml -am

RUN install -m 0755 openssl-dynamic/target/native-lib-only/META-INF/native/linux64/libnetty_tcnative.so /usr/local/lib/libnetty_tcnative.so
~~~

[ctron]

The way to update the OpenSSL lib used is to re-build the Docker image, regardless of whether the OpenSSL lib is maintained as part of the OS image or as part of the layer added by Hono, isn't it?

Not when you are using the S2I images. In this case Hono wouldn't provide new images, but an updates base image (e.g. RHEL) would trigger a new build. Which would re-build the whole stack, but keep the stale static library, containing the bugs.

[ctron]

Well I don't think it is that stupid. It just is the case that C linking and Java don't play well together.

When you build "libssl" on some machine, it does require and enforces the dependencies during run time. Ripping out a compiled library, stuffing it into Maven central and hoping it runs on all target environments is silly from the other viewpoint.

That is why I wouldn't recommend simply taking another fork which runs just fine now, but to really solve the root cause.

[sophokles73]

Can you try replacing the base image with ctron/java-base-tcnative:latest, which is built by:

What's the point of that? Isn't that the same as using one of the (maintained) statically linked versions of tcnative?

[ctron]

What's the point of that? Isn't that the same as using one of the (maintained) statically linked versions of tcnative?

It uses the libssl.so as provided by the OS layer. Debian in this case. So as soon as the Debian image is updated, the libssl.so is updates as well.

My assumption would be that Debian provides quicker updates to libssl.so as a process of BoringSSL adopting changes, then Netty creating a new release on Maven Central and then Hono adopting that new Maven dependency.

[sophokles73]

So as soon as the Debian image is updated, the libssl.so is updates as well.

Actually, that seems to be the exact problem: a distro changing the underlying OpenSSL lib for which it is unclear, if netty-tcnative will still be able to work with ...

Apart from that, are you proposing that we are distributing our own Docker image layer for OpenSSL? If so, we will need to put every version of OpenSSL that we want to use through the IP process, because we would be starting to distribute OpenSSL, right?

[ctron]

Actually, that seems to be the exact problem: a distro changing the underlying OpenSSL lib for which it is unclear, if netty-tcnative will still be able to work with ...

But that is why the build is being used. This will ensure that the native library of tcnative is compiled in the right way with the OS provided libssl.so.

Well right new we already use openjdk images from Dockerhub. Which contains all kinds of dependencies for which we don't have CQs. libssl.so is already present. So we are distributing OpenSSL already. The image I provided does not add OpenSSL.

It is adding netty-tcnative to the image. And I guess we would need a CQ for that. As we don't necessarily require it, it could be a works-with dependency.

On the other hand, if the IP process of Eclipse forces us to ship less secure software, then we should try fixing the IP process. Or maybe that docker image has to live outside of the Hono project. As the current "openjdk" image does.

[sophokles73]

Well right new we already use openjdk images from Dockerhub. Which contains all kinds of dependencies for which we don't have CQs. libssl.so is already present. So we are distributing OpenSSL already. The image I provided does not add OpenSSL.

FMPOV (and I guess the Eclipse Foundation's standpoint as well) we are not distributing Debian (and OpenSSL). We are only distributing a Docker image which is a layer on top of the openjdk layer and that only includes the Hono jars. The user triggers Docker to download the openjdk image by means of starting our image as a container. Docker then under the hood downloads the images and assembles the layer stack ...

However, if our image now contains OpenSSL, then we are in fact distributing OpenSSL ...

[ctron]

However, if our image now contains OpenSSL, then we are in fact distributing OpenSSL ...

The image already contains OpenSSL. I am not adding OpenSSL. It is in the same layer as openjdk.

So I don't see the difference between there.

[sophokles73]

Hmmm, you might be right. In fact, you are not doing an apt install libssl but only an apt install for all the the build deps. What we do install in the new layer is the custom built netty-tcnative. So we would need a CQ for netty-tcnative, I believe, but that shouldn't be a big issue.

[ctron]

Cool. And I agree that the Dockerfile currently is super messy. All those build dependencies should be gone in the end. Doing this all in one step and cleaning up at the same time will reduce the intermediate files and dependencies form the final layer.

But I wanted to check first if that works, before cleaning up.

[sophokles73]

I haven't tried it out yet on my local machine. Let me do so first before going any further, ok?

[Christian-Schmid]

Just FYI: For alpine there is a docker image on github available:

https://github.com/pires/netty-tcnative-alpine

[sophokles73]

@ctron I do not yet see a straightforward way of how I can include the jar file created by the build in your image in the überjar created by the Spring Boot maven plugin during our build process (which runs before the image is being built). We will probably need to run some custom jar commands in order to add it to the überjar.

[ctron]

You don't need to include the JAR. Although the build creates a JAR file, the only important result of the build is the ".so" file copied to "/usr/local/lib". netty-tcnative will try to load either the embedded ".so" or revert to loading a system provided version, which is now in "/usr/local/lib".

[sophokles73]

netty-tcnative itself contains an embedded version of netty_tcnative.so (under META-INF/native). The variants seem to only differ in the way they are linked to OpenSSL. Are you sure that tcnative's Library class first tries to load netty_tcnative.so from /usr/local/lib? I wasn't really able to determine the search order from the source code (yet) ...

[ctron]

Well the search order is not that important. Because if one fails, it still tries the other.

[sophokles73]

what makes you believe that META-INF/native/netty_tcnative.so cannot be loaded?

[ctron]

Depending on which version META-INF/native/netty_tcnative.so you are referring to. The original one on Maven Central due to link dependency issue:

Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libnetty_tcnative8413922882652429363.so: libssl.so.1.0.0: cannot open shared object file: No such file or directory

The one compiled on my test base image can be loaded. That is why it gets copied over to /usr/local/lib where it can be auto-discovered by the library loader.

So when you use the dynamic linked netty-tcnative from Maven Central, without the static additions, then it will load the Java code from it, which tries to load the native library. Which will fail with the embedded version of the .so, but succeed with the one from /usr/local/lib.

[sophokles73]

Ok, so the native library loader seems to first try to load the lib from locations specified in the java.library.path system property (System.getProperty("java.library.path")). However, in my test using your original Dockerfile the Auth service didn't come up successfully. I assume that is because /usr/local/lib is not on the Java library path of the JDK installed on the base image.
However, using the image built with the following Dockerfile

FROM openjdk:jdk-slim

RUN apt-get -y update && apt-get -y dist-upgrade && \
 apt-get -y install wget git autoconf automake libtool libapr1-dev libssl-dev make

RUN wget http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz && \ 
 tar xzf apache-maven-3.5.3-bin.tar.gz

RUN git clone https://github.com/netty/netty-tcnative
WORKDIR netty-tcnative 
RUN git checkout netty-tcnative-parent-2.0.8.Final && \
 /apache-maven-3.5.3/bin/mvn -B clean install -f openssl-dynamic/pom.xml -am

RUN mkdir /usr/lib/jni && \
 install -m 0755 openssl-dynamic/target/native-lib-only/META-INF/native/linux64/libnetty_tcnative.so /usr/lib/jni/libnetty_tcnative.so

I was able to start the Auth service successfully.

However, we also seem to be adding libapr to the original base image so we will also need a CQ for APR as well ...

[ctron]

Cool. Yes libapr.so is, to my understanding, require in any case. As this is the replacement for Java's NIO stack. Either BoringSSL or OpenSSL only provides the TLS layer.

So when it comes to container images, I would really love to have a chat next week with the Eclipse folks and try to get their opinion of what would be part of a release and what not.

Hono itself would IMHO only have a works-with dependency on libapr. As you can still use the plain Java NIO based stack. And netty-tcnative is an alternative. A real improvement, but an alternative.

[sophokles73]

Hono itself would IMHO only have a works-with dependency on libapr.

Not if it is part of the Hono docker image that we distribute.
From the project handbook [1]:

a Works With Dependency should never be directly distributed or otherwise made available without some explicit action from the consumer. The consumer is responsible for making the Works With Dependency content available and otherwise agreeing to the terms for that content.

[1] https://www.eclipse.org/projects/handbook/#ip-third-party-workswith

[ctron]

Just to make myself clear, either way (OpenSSL or static BoringSSL) we would need some kind of coverage for APR. If you look inside the BoringSSL static version .so you will find libapr statically linked inside the dependency:

~
[jreimann@jreimann boring]$ strings META-INF/native/libnetty_tcnative.so | grep apr
apr_thread_mutex_destroy
apr_pool_destroy
apr_thread_mutex_unlock
apr_thread_mutex_lock
apr_pool_create_ex
apr_palloc
apr_pstrdup
apr_thread_mutex_create
apr_os_thread_current
apr_pool_cleanup_null
apr_pool_cleanup_register
apr_version
apr_version_string
apr_initialize
…
~

[ctron]

I know the words … yet I find them completely unrealistic when it comes to container images. So I think that more action (outside of Hono) is needed to solve this.

For this particular issue in Hono, we can wait a week and see if we get better understanding. If that really should be an issue, then the solution would seem rather simple to me. We create a container build which creates the base image. We can distribute that. Then we pick straws and someone uploads this image to dockerhub. And we use that. Because that wouldn't make any difference than the openjdk image IMHO.

[sophokles73]

I see additional drawbacks of this approach:

• the base image will contain the full JDK, not only the JRE as in the image we currently use
• someone (who pulled the shortest straw) needs to be/feel responsible for continuously building/providing/maintaining the base image. If this happens outside of the project itself, I have no real confidence in the stability of this process
• there doesn't seem to be any real advantage of this approach over simply using the statically linked BoringSSL based version apart from the possibility to adopt all changes made to OpenSSL 1.1 very quickly. However, for this to be any real advantage, we would need to start building the Hono images on a nightly basis in an automated way as well (in order to pick up the most recent changes to our custom base image. This leads to the question: do real (i.e. professional) users of Hono will actually use our images at all? Or will they rather build the images themselves anyway in order to also be able to pick up the latest updates to the distro as well? In other words: wouldn't it make more sense to try to distribute Hono's spring boot artifacts via Maven Central and/or Eclipse Repo together with Dockerfiles so that people can more easily build their own images?

[ctron]

the base image will contain the full JDK, not only the JRE as in the image we currently use

I tried to explain this above … this image is not (yet) cleaned up. Once that is done, and I can do this easily, the final image will only contain the JRE and no longer the JDK. See: https://github.com/ctron/kura-emulator/blob/master/Dockerfile

someone (who pulled the shortest straw) needs to be/feel responsible for continuously building/providing/maintaining the base image. If this happens outside of the project itself, I have no real confidence in the stability of this process

The container build would be part of Hono. If that person fails to push changes from the Hono repo (containing the container build) to e.g. docker hub, then a new person will be chosen and that person will do the upload. So I don't see how that should impact the stability of the process. Right now also someone has to do the release process.

This leads to the question: do real (i.e. professional) users of Hono will actually use our images at all?

Going down that road would mean that all images provided by Hono would be insecure by default. And only for testing purposes. So there should be a big warning attached to them. I am not sure that this is the way to do. As I did understand the hawkBit project, which initially intended chose the same strategy, they are now reverting back to actually providing a ready-to-run distribution of hawkBit because that was what the community required.

Building your own images it also doesn't make any sense to me to distribute anything on Maven central. e.g. the S2I build uses the GitHub repos as a base and does a complete build in order to get fresh images once any source component of the final image changes.

However, if Hono ships container images, then I think they should be secure. Or Hono shouldn't ship anything at all. Shipping something which is dangerous to use in production seems like the worst of both worlds to me.

[sophokles73]

I think I have found a more elegant way of adding support for Open/BoringSSL to our Spring Boot application jar on startup without the need to compile in the support.

The Spring Boot bootstrap class for loading the main class and running the application can be configured by means of environment variables to add jars from arbitrary folders to the application's class path [1]

Using this mechanism it is possible to add any of the netty-tcnative variants to the classpath on application startup. When starting the Docker container for one of our services, you could simply mount a config map or other folder containing the netty-tcnative jar you want to use into the container's files system.
I have modified the ConnectionFactoryImpl to probe for the presence of OpenSSL during connection establishment (on branch use_openssl) and have then tried it out using the Example Receiver Spring Boot application adding the static BoringSSL jar like this:

$ cp netty-tcnative-boringssl-static-2.0.7.Final.jar ~/java/lib
$ export LOADER_HOME=/home/kai
$ export LOADER_PATH=java/lib
$ java -jar target/hono-example-0.7-M1-SNAPSHOT-exec.jar --hono.client.host=hono.eclipse.org --hono.client.username=consumer@HONO --hono.client.password=verysecret --spring.profiles.active=receiver --hono.client.tlsEnabled=true

[1] https://docs.spring.io/spring-boot/docs/1.5.13.RELEASE/reference/htmlsingle/#executable-jar-launching

[ctron]

But that would mean we could still also use the before-compiled variant of this as well?!

As we are allowed do our own docker base images, I would try to evolve the OpenShift S2I example deployment in a way that I drop in things like this for all Hono base images. Which doesn't affect the other images/deployments, but provides a custom Hono Java base image when using the S2I approach.

[Christian-Schmid]

Hi,
sorry for the late reply. I like the idea of using the loader.path property very much.
Like this we can use the plain hono images for our deployment and still extend it with e.g. boring ssl.
The PropertiesLauncher worked for me too.

Thanks Chris

[sophokles73]

But that would mean we could still also use the before-compiled variant of this as well?!

Sure, you can still roll your own images. Whether we want to provide images that are pre-configured to use any particular version of OpenSSL is another issue FMPOV. Using the approach outlined above, everybody can use whatever he wants, though.

[sophokles73]

Based on the discussion so far, I think we should use the PropertiesLauncher for Hono services in any case. This will also be useful for later extension like choosing a particular OpenTracing implementation by means of dropping the corresponding jar file into the lib folder.

@Christian-Schmid
I will add support for probing for availability of OpenSSL to Hono's ConnectionFactory and the AbstractServiceBase. You can then rebase #639 and add the config property to require native TLS. However, this will not make it into 0.7-M1 anymore ...

[sophokles73]

@Christian-Schmid I have added my changes, can you rebase #639 and see if everything works for you?

[Christian-Schmid]

@sophokles73: I will have a look at it on monday and give then feedback

[sophokles73]

@Christian-Schmid any news on this?

[Christian-Schmid]

@sophokles73 Sorry for the delay, I rebased now to the current master

[sophokles73]

@Christian-Schmid what about the client side of a TLS connection? Do you also want to require the usage of OpenSSL there as well?

[Christian-Schmid]

Good question :-)

On the server side the wish to use OpenSSL was born because of better performance in scenarios with lots of parallel and short living connections from devices (E.g. single MQTT push and disconnect).

On the client side, the consumers connect to Hono using AMQP 1.0. In general they connect and stay connected for a long time. I guess we wouldn't profit much from the better performance here. (The tough part is during the tls handshake)

So for me the option to enforce it here would just be for aesthetic reasons...

[sophokles73]

@Christian-Schmid you should keep in mind that the protocol adapters all use HonoClients to connect to the registry and the AMQP messaging network, i.e. we are not only talking about devices and applications ...

[Christian-Schmid]

@sophokles73 That is correct. I just wanted to say, that the main benefit of OpenSSL is during the handshake. And handshakes happens way more often on the route device <-> hono adapter than internally where i expect the components to open one amqp connection and then use multiple links.

[sophokles73]

@Christian-Schmid I see. Can you then close this issue, assuming that it is fixed from your point of view?

[ctron]

I think the elegant solution doesn't really work. I wasn't able to get the protocol adapters up and running and dug a little bit into this, maybe I got something wrong … so here is what I think the problem is:

• Spring Boot applications get packaged as "-exec.jar"
• Those JARs use the "JarLauncher": Main-Class: org.springframework.boot.loader.JarLauncher
• According to the Spring Boot documentation the proposed env-vars LOADER_* only work with the PropertiesLauncher

If that is the case then I would propose to actually include netty-tcnative (without a JNI library or BoringSSL) and allow the user to drop in a properly compiled JNI library of his choice. That would work exactly the way it does now and the only thing one needs to do is to provide the "proper" (whatever that means in your OS) JNI library in a "lib" folder.

[sophokles73]

That's why our images run the PropertiesLauncher ...

[ctron]

Well the S2I approach doesn't use those images. It uses "plain Spring Boot". Of course I could re-implement all S2I launcher scripts, but I would prefer to stick to Spring Boot defaults.

Back to my suggestion of adding netty-tcnative as a dependency. In this case we could use default Spring Boot launchers (all of them), we have the same behavior as right now and you could still use your way by using the PropertiesLauncher.

What would speak against that approach?

[sophokles73]

Of course I could re-implement all S2I launcher scripts, but I would prefer to stick to Spring Boot defaults.

I have no insight into S2I, but can't you simply provide the command to invoke for the Docker image that is being built, i.e. the same thing that you would do in a Dockerfile or a Deployment resource descriptor?

[ctron]

Will the S2I Java image does autodetect Spring Boot and simple run the fat JAR. So no customizations are necessary. Of course you can do it all manually, which I would hope not to do.

So again, what would speak against the proposed solution?

[sophokles73]

So again, what would speak against the proposed solution?

Did you check if this works for all scenarios? In particular, I wonder if it works if a user wants to use the BoringSSL jar file. Would the user then add the BoringSSL tcnative jar to the classpath using the approach described above and would the system be able to detect the JNI lib contained in the additional jar?

[ctron]

I would wonder who would like to use BoringSSL … But yes, netty-tcnative loads the JNI wrapper from a bunch of locations, including the classpath.

But if that helps I will try to come up with an automatic test for this scenario.

[ctron]

Commit dcae26c12f6db4f85f061d76ae175fd6ec22ad0c adds a validation for BoringSSL to the integration tests. So adding netty-tcnative to Hono as a regular dependency still allows you to use the PropertiesLauncher in the way it currently does.

I created PR #698 which shows how this works with S2I, adding netty-tcnative to Hono and adding the test in the integration tests. Does that work for you?

[sophokles73]

Let me try to get this straight. The idea now would be to include the standard netty-tcnative jar in the Spring Boot application jar. During startup

1. tcnative first tries to load a dynamically linked version of OpenSSL that exists on the operating system
2. if not found, it tries to load a statically linked version of OpenSSL embedded into a jar file on the class file

So, if you run the Spring Boot application on the Docker image that you built using S2I then the OpenSSL version installed on the OS will be used when you run the application using the standard JarLauncher. But if you run the application on Hono's standard Docker image, then OpenSSL is only used, if you have added a statically linked version in a jar to the extension class path and start the application using the PropertiesLauncher.

Is this correct?

[ctron]

Mostly. … The Java part of netty-tcnative does not load OpenSSL directly. It only loads its JNI wrapper library.

By default after the PR is merged the following will happen:

1. tcnative tries to load its JNI wrapper but doesn't find it in the standard Hono images
2. It falls back to JDK TLS, if the "native required" flag is set it will fail

For the S2I images the following will happen:

1. tcnative tries to load its JNI wrapper and finds it in the LD_LIBRARY_PATH
2. When the JNI wrapper is loaded it will automatically load the libssl.so using pure operating system mechanisms.

For Hono with the PropertiesLauncher flags set to include netty-tcnative-boringssl:

1. tcnative tries to load its JNI wrapper and finds it on the classpath
2. The JNI wrapper already contains "BoringSSL" statically linked into the JNI wrapper and nothing else will be loaded

This all results into the possibility that you can provide the JNI in any possible way you like (OS layer or classpath). As long as it is present and loadable, it can be used.

[sophokles73]

I have created #713 for adding netty-tcnative