Quarkus: error on ssl handshake with database in native image mode

[haraldatbmw]

Describe the bug

• I have a quarkus application which makes a flyway database migration at startup.

• My database is a postgres db allowing only ssl connections.

• I compile a native image successfully.

[INFO] --- quarkus-maven-plugin:0.25.0:build (default) @ instantmobile ---
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Beginning quarkus augmentation
[INFO] [org.jboss.threads] JBoss Threads version 3.0.0.Final
[INFO] [io.quarkus.flyway.FlywayProcessor] Adding application migrations in path '/C:/git/Trend-Radar/instantmobile-quarkus/target/classes/db/migration' using protocol 'file'
[INFO] [org.hibernate.jpa.boot.internal.PersistenceXmlParser] HHH000318: Could not find any META-INF/persistence.xml file in the classpath
[INFO] [org.hibernate.Version] HHH000412: Hibernate Core {5.4.6.Final}
[INFO] [io.quarkus.resteasy] Resteasy running without servlet container.
[INFO] [io.quarkus.resteasy] - Add quarkus-undertow to run Resteasy within a servlet container
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 3214ms
[INFO] [io.quarkus.creator.phase.runnerjar.RunnerJarPhase] Building jar: C:\git\Trend-Radar\instantmobile-quarkus\target\instantmobile-1.0-SNAPSHOT-runner.jar
[INFO]
[INFO] --- quarkus-maven-plugin:0.25.0:native-image (default) @ instantmobile ---
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] Running Quarkus native-image plugin on OpenJDK 64-Bit Server VM
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] docker run -v C:\git\Trend-Radar\instantmobile-quarkus\target:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.0.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.netty.leakDetection.level=DISABLED -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dio.netty.allocator.maxOrder=1 -J-Dvertx.disableDnsResolver=true -H:IncludeResources=META-INF/resources/.* --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar instantmobile-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+PrintAnalysisCallTree -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services -H:+JNI -H:-UseServiceLoaderFeature -H:+StackTrace
Build on Server(pid: 28, port: 36615)*
[instantmobile-1.0-SNAPSHOT-runner:28]    classlist:  16,022.07 ms
[instantmobile-1.0-SNAPSHOT-runner:28]        (cap):   1,483.57 ms
[instantmobile-1.0-SNAPSHOT-runner:28]        setup:   3,468.05 ms
13:14:36,465 INFO  [org.hib.Version] HHH000412: Hibernate Core {5.4.6.Final}
13:14:36,492 INFO  [org.hib.ann.com.Version] HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
13:14:36,533 INFO  [org.hib.dia.Dialect] HHH000400: Using dialect: io.quarkus.hibernate.orm.runtime.dialect.QuarkusPostgreSQL95Dialect
13:14:38,822 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Final
13:14:40,077 INFO  [com.arj.ats.arjuna] ARJUNA012170: TransactionStatusManager started on port 36521 and host 127.0.0.1 with service com.arjuna.ats.arjuna.recovery.ActionStatusService
[instantmobile-1.0-SNAPSHOT-runner:28]   (typeflow):  53,230.37 ms
[instantmobile-1.0-SNAPSHOT-runner:28]    (objects):  22,599.06 ms
[instantmobile-1.0-SNAPSHOT-runner:28]   (features):   1,261.27 ms
[instantmobile-1.0-SNAPSHOT-runner:28]     analysis:  81,527.13 ms
Printing call tree to /project/reports/call_tree_instantmobile-1.0-SNAPSHOT-runner_20191017_131619.txt
Printing list of used classes to /project/reports/used_classes_instantmobile-1.0-SNAPSHOT-runner_20191017_131630.txt
Printing list of used packages to /project/reports/used_packages_instantmobile-1.0-SNAPSHOT-runner_20191017_131630.txt
[instantmobile-1.0-SNAPSHOT-runner:28]     (clinit):   1,202.76 ms
[instantmobile-1.0-SNAPSHOT-runner:28]     universe:   3,882.22 ms
[instantmobile-1.0-SNAPSHOT-runner:28]      (parse):   9,987.58 ms
[instantmobile-1.0-SNAPSHOT-runner:28]     (inline):   9,373.37 ms
[instantmobile-1.0-SNAPSHOT-runner:28]    (compile):  57,773.18 ms
[instantmobile-1.0-SNAPSHOT-runner:28]      compile:  80,990.49 ms
[instantmobile-1.0-SNAPSHOT-runner:28]        image:   4,884.87 ms
[instantmobile-1.0-SNAPSHOT-runner:28]        write:   2,377.36 ms
[instantmobile-1.0-SNAPSHOT-runner:28]      [total]: 227,369.27 ms
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:17 min
[INFO] Finished at: 2019-10-17T15:18:05+02:00
[INFO] ------------------------------------------------------------------------

• If i run the application it fails with the following error.

harald@xxx:/instantmobile-quarkus/target$ ./instantmobile-1.0-SNAPSHOT-runner -Djava.library.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64 -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts

13:29:05 INFO  [org.flywaydb.core.internal.license.VersionPrinter]] (main) Flyway Community Edition 6.0.4 by Redgate
WARNING: The sunec native library, required by the SunEC provider, could not be loaded. This library is usually shipped as part of the JDK and can be found under
<JAVA_HOME>/jre/lib/<platform>/libsunec.so. It is loaded at run time via System.loadLibrary("sunec"), the first time services from SunEC are accessed. To use this provider's services the java.library.path system property needs to be set accordingly to point to a location that contains libsunec.so. Note that if java.library.path is not set it defaults to the current working directory.
java.lang.UnsatisfiedLinkError: sun.security.ec.ECKeyPairGenerator.isCurveSupported([B)Z [symbol: Java_sun_security_ec_ECKeyPairGenerator_isCurveSupported or Java_sun_security_ec_ECKeyPairGenerator_isCurveSupported___3B]
        at com.oracle.svm.jni.access.JNINativeLinkage.getOrFindEntryPoint(JNINativeLinkage.java:145)
        at com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallAddress(JNIGeneratedMethodSupport.java:57)
        at sun.security.ec.ECKeyPairGenerator.isCurveSupported(ECKeyPairGenerator.java)
        at sun.security.ec.ECKeyPairGenerator.ensureCurveIsSupported(ECKeyPairGenerator.java:135)
        at sun.security.ec.ECKeyPairGenerator.initialize(ECKeyPairGenerator.java:114)
        at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:674)
        at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:77)
        at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:783)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:302)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:40)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:441)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:94)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:458)
        at org.postgresql.Driver.connect(Driver.java:260)
        at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:200)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:370)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:352)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Exception in thread "main" java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl1.doStart(ApplicationImpl1.zig:268)
        at io.quarkus.runtime.Application.start(Application.java:94)
        at io.quarkus.runtime.Application.run(Application.java:218)
        at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
Caused by: java.lang.UnsatisfiedLinkError: sun.security.ec.ECKeyPairGenerator.isCurveSupported([B)Z [symbol: Java_sun_security_ec_ECKeyPairGenerator_isCurveSupported or Java_sun_security_ec_ECKeyPairGenerator_isCurveSupported___3B]
        at com.oracle.svm.jni.access.JNINativeLinkage.getOrFindEntryPoint(JNINativeLinkage.java:145)
        at com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallAddress(JNIGeneratedMethodSupport.java:57)
        at sun.security.ec.ECKeyPairGenerator.isCurveSupported(ECKeyPairGenerator.java)
        at sun.security.ec.ECKeyPairGenerator.ensureCurveIsSupported(ECKeyPairGenerator.java:135)
        at sun.security.ec.ECKeyPairGenerator.initialize(ECKeyPairGenerator.java:114)
        at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:674)
        at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:77)
        at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:783)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:302)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:40)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:441)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:94)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:458)
        at org.postgresql.Driver.connect(Driver.java:260)
        at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:200)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:370)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:352)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
harald@xxx:/instantmobile-quarkus/target$

• Same error happens when building and running a docker image from the native image.
• Same error happens when building and running inside openshift with native image.
• I tried different ways to point to the libsunec.so without success.
  
  
  
  • point java.library.path to JRE lib directory (see log)
  
  
  • copy libsunec.so next to the application
  
  
  • point java.library.path direct to the filename
  
  

Expected behavior
Successful SSL connection to my postgres database.

Actual behavior
Can not start to application due to ssl handshake error.

To Reproduce
Steps to reproduce the behavior:

1. Create quarkus application with flyway and postgres jdbc
2. Host postgres database with ssl connectivity
3. Build and run a native image of the application

Environment (please complete the following information):

• Output of uname -a or ver:
  Linux LMUC795407 4.4.0-17134-Microsoft #706-Microsoft Mon Apr 01 18:13:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

• Output of java -version:
  openjdk version "1.8.0_191"
  OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)
  OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

• Quarkus version or git rev: 0.25.0

[jaikiran]

./instantmobile-1.0-SNAPSHOT-runner -Djava.library.path=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64 -Djavax.net.ssl.trustStore=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts

Can you post the output of:

ls -lh /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64 

or even the output of:

tree /usr/lib/jvm/java-8-openjdk-amd64/jre

[haraldatbmw]

Output of jre/lib/amd64:

harald@xxx:/instantmobile-quarkus$ ls -lh /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
total 4.2M
drwxr-xr-x 1 root root  4.0K Apr 16  2019 jli
lrwxrwxrwx 1 root root    33 Jan 14  2019 jvm.cfg -> /etc/java-8-openjdk/jvm-amd64.cfg
-rw-r--r-- 1 root root   278 Jan 14  2019 jvm.cfg-default
-rw-r--r-- 1 root root   15K Jan 14  2019 libattach.so
-rw-r--r-- 1 root root  685K Jan 14  2019 libawt.so
-rw-r--r-- 1 root root   31K Jan 14  2019 libawt_headless.so
-rw-r--r-- 1 root root  348K Jan 14  2019 libawt_xawt.so
-rw-r--r-- 1 root root   19K Jan 14  2019 libdt_socket.so
-rw-r--r-- 1 root root  336K Jan 14  2019 libfontmanager.so
-rw-r--r-- 1 root root  187K Jan 14  2019 libhprof.so
-rw-r--r-- 1 root root   51K Jan 14  2019 libicedtea-sound.so
-rw-r--r-- 1 root root   43K Jan 14  2019 libinstrument.so
-rw-r--r-- 1 root root   39K Jan 14  2019 libj2gss.so
-rw-r--r-- 1 root root   15K Jan 14  2019 libj2pcsc.so
-rw-r--r-- 1 root root   67K Jan 14  2019 libj2pkcs11.so
-rw-r--r-- 1 root root  6.1K Jan 14  2019 libjaas_unix.so
-rw-r--r-- 1 root root  191K Jan 14  2019 libjava.so
-rw-r--r-- 1 root root   23K Jan 14  2019 libjava_crw_demo.so
-rw-r--r-- 1 root root   47K Jan 14  2019 libjavajpeg.so
-rw-r--r-- 1 root root   19K Jan 14  2019 libjavalcms.so
-rw-r--r-- 1 root root  6.0K Jan 14  2019 libjawt.so
-rw-r--r-- 1 root root  240K Jan 14  2019 libjdwp.so
-rw-r--r-- 1 root root   11K Jan 14  2019 libjsdt.so
-rw-r--r-- 1 root root   11K Jan 14  2019 libjsig.so
-rw-r--r-- 1 root root  6.0K Jan 14  2019 libjsound.so
-rw-r--r-- 1 root root   64K Jan 14  2019 libjsoundalsa.so
-rw-r--r-- 1 root root   39K Jan 14  2019 libmanagement.so
-rw-r--r-- 1 root root 1003K Jan 14  2019 libmlib_image.so
-rw-r--r-- 1 root root   95K Jan 14  2019 libnet.so
-rw-r--r-- 1 root root   75K Jan 14  2019 libnio.so
-rw-r--r-- 1 root root   15K Jan 14  2019 libnpt.so
-rw-r--r-- 1 root root   51K Jan 14  2019 libsaproc.so
-rw-r--r-- 1 root root   23K Jan 14  2019 libsctp.so
-rw-r--r-- 1 root root   43K Jan 14  2019 libsplashscreen.so
-rw-r--r-- 1 root root  179K Jan 14  2019 libsunec.so
-rw-r--r-- 1 root root   98K Jan 14  2019 libunpack.so
-rw-r--r-- 1 root root   71K Jan 14  2019 libverify.so
-rw-r--r-- 1 root root   35K Jan 14  2019 libzip.so
drwxr-xr-x 1 root root  4.0K Apr 16  2019 server

And the tree output:

harald@xxx:/instantmobile-quarkus$ tree /usr/lib/jvm/java-8-openjdk-amd64/jre
/usr/lib/jvm/java-8-openjdk-amd64/jre
├── ASSEMBLY_EXCEPTION
├── THIRD_PARTY_README
├── bin
│   ├── java
│   ├── jjs
│   ├── keytool
│   ├── orbd
│   ├── pack200
│   ├── policytool
│   ├── rmid
│   ├── rmiregistry
│   ├── servertool
│   ├── tnameserv
│   └── unpack200
├── lib
│   ├── accessibility.properties -> /etc/java-8-openjdk/accessibility.properties
│   ├── amd64
│   │   ├── jli
│   │   │   └── libjli.so
│   │   ├── jvm.cfg -> /etc/java-8-openjdk/jvm-amd64.cfg
│   │   ├── jvm.cfg-default
│   │   ├── libattach.so
│   │   ├── libawt.so
│   │   ├── libawt_headless.so
│   │   ├── libawt_xawt.so
│   │   ├── libdt_socket.so
│   │   ├── libfontmanager.so
│   │   ├── libhprof.so
│   │   ├── libicedtea-sound.so
│   │   ├── libinstrument.so
│   │   ├── libj2gss.so
│   │   ├── libj2pcsc.so
│   │   ├── libj2pkcs11.so
│   │   ├── libjaas_unix.so
│   │   ├── libjava.so
│   │   ├── libjava_crw_demo.so
│   │   ├── libjavajpeg.so
│   │   ├── libjavalcms.so
│   │   ├── libjawt.so
│   │   ├── libjdwp.so
│   │   ├── libjsdt.so
│   │   ├── libjsig.so
│   │   ├── libjsound.so
│   │   ├── libjsoundalsa.so
│   │   ├── libmanagement.so
│   │   ├── libmlib_image.so
│   │   ├── libnet.so
│   │   ├── libnio.so
│   │   ├── libnpt.so
│   │   ├── libsaproc.so
│   │   ├── libsctp.so
│   │   ├── libsplashscreen.so
│   │   ├── libsunec.so
│   │   ├── libunpack.so
│   │   ├── libverify.so
│   │   ├── libzip.so
│   │   └── server
│   │       ├── Xusage.txt
│   │       ├── classes.jsa
│   │       ├── libjsig.so -> ../libjsig.so
│   │       └── libjvm.so
│   ├── calendars.properties -> /etc/java-8-openjdk/calendars.properties
│   ├── charsets.jar
│   ├── classlist
│   ├── cmm
│   │   ├── CIEXYZ.pf
│   │   ├── GRAY.pf
│   │   ├── LINEAR_RGB.pf
│   │   ├── PYCC.pf
│   │   └── sRGB.pf
│   ├── content-types.properties -> /etc/java-8-openjdk/content-types.properties
│   ├── currency.data
│   ├── ext
│   │   ├── cldrdata.jar
│   │   ├── dnsns.jar
│   │   ├── icedtea-sound.jar
│   │   ├── jaccess.jar
│   │   ├── java-atk-wrapper.jar -> ../../../../../../share/java/java-atk-wrapper.jar
│   │   ├── libatk-wrapper.so -> ../../../../../x86_64-linux-gnu/jni/libatk-wrapper.so
│   │   ├── localedata.jar
│   │   ├── nashorn.jar
│   │   ├── sunec.jar
│   │   ├── sunjce_provider.jar
│   │   ├── sunpkcs11.jar
│   │   └── zipfs.jar
│   ├── flavormap.properties -> /etc/java-8-openjdk/flavormap.properties
│   ├── hijrah-config-umalqura.properties
│   ├── images
│   │   └── cursors
│   │       ├── cursors.properties -> /etc/java-8-openjdk/images/cursors/cursors.properties
│   │       ├── invalid32x32.gif
│   │       ├── motif_CopyDrop32x32.gif
│   │       ├── motif_CopyNoDrop32x32.gif
│   │       ├── motif_LinkDrop32x32.gif
│   │       ├── motif_LinkNoDrop32x32.gif
│   │       ├── motif_MoveDrop32x32.gif
│   │       └── motif_MoveNoDrop32x32.gif
│   ├── jar.binfmt
│   ├── jce.jar
│   ├── jexec
│   ├── jsse.jar
│   ├── jvm.hprof.txt
│   ├── logging.properties -> /etc/java-8-openjdk/logging.properties
│   ├── management
│   │   ├── jmxremote.access -> /etc/java-8-openjdk/management/jmxremote.access
│   │   ├── jmxremote.password -> /etc/java-8-openjdk/management/jmxremote.password
│   │   ├── management.properties -> /etc/java-8-openjdk/management/management.properties
│   │   └── snmp.acl -> /etc/java-8-openjdk/management/snmp.acl
│   ├── management-agent.jar
│   ├── meta-index
│   ├── net.properties -> /etc/java-8-openjdk/net.properties
│   ├── psfont.properties.ja -> /etc/java-8-openjdk/psfont.properties.ja
│   ├── psfontj2d.properties -> /etc/java-8-openjdk/psfontj2d.properties
│   ├── resources.jar
│   ├── rt.jar
│   ├── security
│   │   ├── blacklisted.certs -> /etc/java-8-openjdk/security/blacklisted.certs
│   │   ├── cacerts -> /etc/ssl/certs/java/cacerts
│   │   ├── java.policy -> /etc/java-8-openjdk/security/java.policy
│   │   ├── java.security -> /etc/java-8-openjdk/security/java.security
│   │   ├── nss.cfg -> /etc/java-8-openjdk/security/nss.cfg
│   │   └── policy
│   │       ├── limited
│   │       │   ├── US_export_policy.jar
│   │       │   └── local_policy.jar
│   │       └── unlimited
│   │           ├── US_export_policy.jar
│   │           └── local_policy.jar
│   ├── sound.properties -> /etc/java-8-openjdk/sound.properties
│   ├── swing.properties -> /etc/java-8-openjdk/swing.properties
│   └── tzdb.dat
└── man
    ├── ja -> ja_JP.UTF-8
    ├── ja_JP.UTF-8
    │   └── man1
    │       ├── java.1.gz
    │       ├── jjs.1.gz
    │       ├── keytool.1.gz
    │       ├── orbd.1.gz
    │       ├── pack200.1.gz
    │       ├── policytool.1.gz
    │       ├── rmid.1.gz
    │       ├── rmiregistry.1.gz
    │       ├── servertool.1.gz
    │       ├── tnameserv.1.gz
    │       └── unpack200.1.gz
    └── man1
        ├── java.1.gz
        ├── jjs.1.gz
        ├── keytool.1.gz
        ├── orbd.1.gz
        ├── pack200.1.gz
        ├── policytool.1.gz
        ├── rmid.1.gz
        ├── rmiregistry.1.gz
        ├── servertool.1.gz
        ├── tnameserv.1.gz
        └── unpack200.1.gz

19 directories, 139 files

[jaikiran]

Thank you. While I have you here, can you also post us the output of:

echo $GRAALVM_HOME

and then

ls -lh $GRAALVM_HOME/jre/lib/amd64

and

tree $GRAALVM_HOME/jre/

[haraldatbmw]

Just reinstalled the latest graalvm for linux (befor my runtime was openjdk only) but resulted in the same error.

My new call of my executable:

./instantmobile-1.0-SNAPSHOT-runner -Djava.library.path=$GRAALVM_HOME/jre/lib/amd64 -Djavax.net.ssl.trustStore=$GRAALVM_HOME/jre/lib/security/cacerts

My graalvm env variable:

harald@xxx:/instantmobile-quarkus/target$ echo $GRAALVM_HOME
/home/harald/graalvm/graalvm-ce-19.2.1

The libraries:

harald@xxx:/instantmobile-quarkus/target$ ls -lh $GRAALVM_HOME/jre/lib/amd64
total 37M
drwxrwxrwx 1 harald harald 4.0K Oct 18 12:52 jli
-rw-r--r-- 1 harald harald 1.6K Oct 12 02:22 jvm.cfg
-rwxr-xr-x 1 harald harald  19K Oct  8 12:50 libattach.so
-rwxr-xr-x 1 harald harald 758K Oct  8 12:50 libawt.so
-rwxr-xr-x 1 harald harald  40K Oct  8 12:50 libawt_headless.so
-rwxr-xr-x 1 harald harald 480K Oct  8 12:50 libawt_xawt.so
-rwxr-xr-x 1 harald harald  25K Oct  8 12:50 libdt_socket.so
-rwxr-xr-x 1 harald harald 533K Oct  8 12:50 libfontmanager.so
-rwxr-xr-x 1 harald harald 208K Oct  8 12:50 libhprof.so
-rwxr-xr-x 1 harald harald  52K Oct  8 12:50 libinstrument.so
-rwxr-xr-x 1 harald harald  47K Oct  8 12:50 libj2gss.so
-rwxr-xr-x 1 harald harald  19K Oct  8 12:50 libj2pcsc.so
-rwxr-xr-x 1 harald harald  89K Oct  8 12:50 libj2pkcs11.so
-rwxr-xr-x 1 harald harald 8.2K Oct  8 12:50 libjaas_unix.so
-rwxr-xr-x 1 harald harald 207K Oct  8 12:50 libjava.so
-rwxr-xr-x 1 harald harald  26K Oct  8 12:50 libjava_crw_demo.so
-rwxr-xr-x 1 harald harald 8.2K Oct  8 12:50 libjawt.so
-rwxr-xr-x 1 harald harald 268K Oct  8 12:50 libjdwp.so
-rwxr-xr-x 1 harald harald 230K Oct  8 12:50 libjpeg.so
-rwxr-xr-x 1 harald harald  13K Oct  8 12:50 libjsdt.so
-rwxr-xr-x 1 harald harald  11K Oct 12 02:23 libjsig.so
-rwxr-xr-x 1 harald harald 8.4K Oct  8 12:50 libjsound.so
-rwxr-xr-x 1 harald harald  83K Oct  8 12:50 libjsoundalsa.so
-rwxr-xr-x 1 harald harald  31M Oct 12 17:16 libjvmcicompiler.so
-rwxr-xr-x 1 harald harald 421K Oct  8 12:50 liblcms.so
-rwxr-xr-x 1 harald harald  52K Oct  8 12:50 libmanagement.so
-rwxr-xr-x 1 harald harald 895K Oct  8 12:50 libmlib_image.so
-rwxr-xr-x 1 harald harald 116K Oct  8 12:50 libnet.so
-rwxr-xr-x 1 harald harald  94K Oct  8 12:50 libnio.so
-rwxr-xr-x 1 harald harald  18K Oct  8 12:50 libnpt.so
-rwxr-xr-x 1 harald harald  50K Oct 12 02:23 libsaproc.so
-rwxr-xr-x 1 harald harald  30K Oct  8 12:50 libsctp.so
-rwxr-xr-x 1 harald harald 443K Oct  8 12:50 libsplashscreen.so
-rwxr-xr-x 1 harald harald 254K Oct  8 12:50 libsunec.so
-rwxr-xr-x 1 harald harald 250K Oct 12 16:32 libtrufflenfi.so
-rwxr-xr-x 1 harald harald 165K Oct  8 12:50 libunpack.so
-rwxr-xr-x 1 harald harald  73K Oct  8 12:50 libverify.so
-rwxr-xr-x 1 harald harald 129K Oct  8 12:50 libzip.so
drwxrwxrwx 1 harald harald 4.0K Oct 18 12:52 server

The directory tree:

harald@xxx:~$ tree $GRAALVM_HOME/jre/lib
/home/harald/graalvm/graalvm-ce-19.2.1/jre/lib
├── amd64
│   ├── jli
│   │   └── libjli.so
│   ├── jvm.cfg
│   ├── libattach.so
│   ├── libawt.so
│   ├── libawt_headless.so
│   ├── libawt_xawt.so
│   ├── libdt_socket.so
│   ├── libfontmanager.so
│   ├── libhprof.so
│   ├── libinstrument.so
│   ├── libj2gss.so
│   ├── libj2pcsc.so
│   ├── libj2pkcs11.so
│   ├── libjaas_unix.so
│   ├── libjava.so
│   ├── libjava_crw_demo.so
│   ├── libjawt.so
│   ├── libjdwp.so
│   ├── libjpeg.so
│   ├── libjsdt.so
│   ├── libjsig.so
│   ├── libjsound.so
│   ├── libjsoundalsa.so
│   ├── libjvmcicompiler.so
│   ├── liblcms.so
│   ├── libmanagement.so
│   ├── libmlib_image.so
│   ├── libnet.so
│   ├── libnio.so
│   ├── libnpt.so
│   ├── libsaproc.so
│   ├── libsctp.so
│   ├── libsplashscreen.so
│   ├── libsunec.so
│   ├── libtrufflenfi.so
│   ├── libunpack.so
│   ├── libverify.so
│   ├── libzip.so
│   └── server
│       ├── Xusage.txt
│       ├── libjsig.so
│       ├── libjvm.so
│       └── vm.properties
├── boot
│   ├── graal-sdk.jar
│   ├── graal-sdk.src.zip
│   └── graaljs-scriptengine.jar
├── calendars.properties
├── charsets.jar
├── classlist
├── cmm
│   ├── CIEXYZ.pf
│   ├── GRAY.pf
│   ├── LINEAR_RGB.pf
│   ├── PYCC.pf
│   └── sRGB.pf
├── content-types.properties
├── currency.data
├── ext
│   ├── cldrdata.jar
│   ├── dnsns.jar
│   ├── jaccess.jar
│   ├── localedata.jar
│   ├── meta-index
│   ├── nashorn.jar
│   ├── sunec.jar
│   ├── sunjce_provider.jar
│   ├── sunpkcs11.jar
│   └── zipfs.jar
├── flavormap.properties
├── graal
│   ├── graal-compiler-match-processor.jar
│   ├── graal-nodeinfo-processor.jar
│   ├── graal-options-processor.jar
│   ├── graal-processor-common.jar
│   ├── graal-replacements-processor.jar
│   └── graal-serviceprovider-processor.jar
├── graalvm
│   ├── graal-hotspot-library.jar
│   ├── graal-truffle-compiler-libgraal.jar
│   ├── graaljs-launcher.jar
│   ├── launcher-common.jar
│   ├── launcher-common.src.zip
│   ├── sulong-launcher.jar
│   └── sulong-toolchain-launchers.jar
├── hijrah-config-umalqura.properties
├── images
│   └── cursors
│       ├── cursors.properties
│       ├── invalid32x32.gif
│       ├── motif_CopyDrop32x32.gif
│       ├── motif_CopyNoDrop32x32.gif
│       ├── motif_LinkDrop32x32.gif
│       ├── motif_LinkNoDrop32x32.gif
│       ├── motif_MoveDrop32x32.gif
│       └── motif_MoveNoDrop32x32.gif
├── jce.jar
├── jexec
├── jsse.jar
├── jvm.hprof.txt
├── jvmci
│   ├── compiler-name
│   ├── graal-management.jar
│   ├── graal.jar
│   ├── jvmci-api.jar
│   ├── jvmci-api.src.zip
│   ├── jvmci-hotspot.jar
│   ├── jvmci-hotspot.src.zip
│   └── parentClassLoader.classpath
├── jvmci-services.jar
├── jvmci-services.src.zip
├── libfdlibm.a
├── libjaas.a
├── libjava.a
├── libnet.a
├── libnio.a
├── libsunec.a
├── libzip.a
├── logging.properties
├── management
│   ├── jmxremote.access
│   ├── jmxremote.password.template
│   ├── management.properties
│   └── snmp.acl.template
├── management-agent.jar
├── meta-index
├── net.properties
├── polyglot
│   ├── graal_isolate.h
│   ├── graal_isolate_dynamic.h
│   ├── libpolyglot.h
│   ├── libpolyglot.so
│   ├── libpolyglot_dynamic.h
│   ├── polyglot-native-api.jar
│   ├── polyglot_api.h
│   ├── polyglot_api_dynamic.h
│   ├── polyglot_isolate.h
│   ├── polyglot_isolate_dynamic.h
│   └── polyglot_types.h
├── psfont.properties.ja
├── psfontj2d.properties
├── resources.jar
├── rt.jar
├── security
│   ├── blacklisted.certs
│   ├── cacerts
│   ├── java.policy
│   ├── java.security
│   └── policy
│       ├── limited
│       │   ├── US_export_policy.jar
│       │   └── local_policy.jar
│       └── unlimited
│           ├── US_export_policy.jar
│           └── local_policy.jar
├── sound.properties
├── svm
│   ├── builder
│   │   ├── clibraries
│   │   │   └── linux-amd64
│   │   │       ├── include
│   │   │       │   ├── cpufeatures.h
│   │   │       │   ├── ffi.h
│   │   │       │   ├── ffitarget.h
│   │   │       │   ├── svm_libffi.h
│   │   │       │   └── trufflenfi.h
│   │   │       ├── libffi.a
│   │   │       ├── libjvm.a
│   │   │       ├── liblibchelper.a
│   │   │       └── libstrictmath.a
│   │   ├── graal-llvm.jar
│   │   ├── javacpp.jar
│   │   ├── llvm-platform-specific.jar
│   │   ├── llvm-wrapper.jar
│   │   ├── objectfile.jar
│   │   ├── pointsto.jar
│   │   ├── svm-llvm.jar
│   │   └── svm.jar
│   ├── clibraries
│   │   └── linux-amd64
│   │       ├── include
│   │       │   ├── cpufeatures.h
│   │       │   ├── ffi.h
│   │       │   ├── ffitarget.h
│   │       │   ├── svm_libffi.h
│   │       │   └── trufflenfi.h
│   │       ├── libffi.a
│   │       ├── libjvm.a
│   │       ├── liblibchelper.a
│   │       └── libstrictmath.a
│   ├── library-support.jar
│   └── macros
│       ├── graalvm-native-clang++-launcher
│       │   └── native-image.properties
│       ├── graalvm-native-clang-launcher
│       │   └── native-image.properties
│       ├── gu-launcher
│       │   └── native-image.properties
│       ├── js-launcher
│       │   ├── native-image.properties
│       │   └── polyglot.config
│       ├── jvmcicompiler-library
│       │   └── native-image.properties
│       ├── lli-launcher
│       │   ├── native-image.properties
│       │   └── polyglot.config
│       ├── polyglot-launcher
│       │   └── native-image.properties
│       ├── polyglot-library
│       │   └── native-image.properties
│       └── truffle
│           └── native-image.properties
├── truffle
│   ├── locator.jar
│   ├── locator.src.zip
│   ├── truffle-api.jar
│   ├── truffle-api.src.zip
│   ├── truffle-dsl-processor.jar
│   ├── truffle-dsl-processor.src.zip
│   ├── truffle-tck.jar
│   └── truffle-tck.src.zip
└── tzdb.dat

36 directories, 188 files

[jaikiran]

I had a bit more closer look at your logs. So:

[INFO] [io.quarkus.creator.phase.runnerjar.RunnerJarPhase] Building jar: C:\git\Trend-Radar\instantmobile-quarkus\target\instantmobile-1.0-SNAPSHOT-runner.jar
[INFO]

So you are building the native-image on Windows OS? Of course using docker. And then copying over that generated native-image to some other (linux?) system and running/launching it from there?

[jaikiran]

By the way, does that exception occur when you hit some rest endpoint? If so, we can sort this out very quickly if you can add the following statement as the first line in your rest endpoint method, before doing the other implementation:

System.out.println("Java library path is set to " + System.getProperty("java.library.path"));

If you can get that output log when you launch your application, then we will know what exactly is going on.

[haraldatbmw]

Some infos about my environment:

• My operating system is Windows 10
• I compiled the native image using docker
• On my Windows 10 I have installed the "Linux Subsystem for Windows" to run the native executable

I did an additional successful test (without docker build):

• I installed GraalVM and native-image to my "Linux Subsystem for Windows" (WSL)
• I built the native image without docker in WSL
• I was able to start the application without an SSL error

So for me it seams to be an issue with the docker build.
The error only occurs if the native image, built with the docker build, is started.

You asked me when the exception occurs - it occurs on application startup (no REST endpoint).
Flyway db migration is automatically started at application startup and tries to migrate the database.

[haraldatbmw]

@jaikiran Are you able to reproduce the issue?

[haraldatbmw]

I have an other running solution, using a multistage docker build.
This works without any errors.

## Stage 1 : build with maven builder image with native capabilities
FROM quay.io/quarkus/centos-quarkus-maven:19.2.1 AS build
COPY src /usr/src/app/src
COPY pom.xml /usr/src/app
COPY settings.xml /usr/src/app
USER root
RUN chown -R quarkus /usr/src/app
USER quarkus
RUN mvn -f /usr/src/app/pom.xml -s /usr/src/app/settings.xml -Pnative clean package

## Stage 2 : ssl-libs
FROM quay.io/quarkus/ubi-quarkus-native-image:19.2.1 as nativebuilder
RUN mkdir -p /tmp/ssl-libs/lib \
  && cp /opt/graalvm/jre/lib/security/cacerts /tmp/ssl-libs \
  && cp /opt/graalvm/jre/lib/amd64/libsunec.so /tmp/ssl-libs/lib/

## Stage 3 : create the docker final image
FROM registry.access.redhat.com/ubi8/ubi-minimal
WORKDIR /work/
COPY --from=build /usr/src/app/target/*-runner /work/application
COPY --from=nativebuilder /tmp/ssl-libs/ /work/
RUN chmod 775 /work
EXPOSE 8080
CMD ["./application", "-Dquarkus.http.host=0.0.0.0", "-Djava.library.path=/work/lib", "-Djavax.net.ssl.trustStore=/work/cacerts"]

[haraldatbmw]

So the error remaining is using the native image docker build and then building and running the docker-image or running it directly inside WSL in Windows10.

mvn package -Pnative -Dnative-image.docker-build=true
docker build -f src/main/docker/Dockerfile.native -t quarkus/instantmobile .
docker run -i --rm -p 8080:8080 --name instantmobile quarkus/instantmobile

[jaikiran]

@haraldatbmw Thank you for the detailed information. I have been able to correlate this back to the code which sets up this infrastructure. I do see that there is an issue in the way this is done. However, I don't see any other way out, other than probably adding a prominent note explain these issues in the guide. Before getting to that though, I want to make sure I've understood the code correctly, so I'm just waiting for some spare time that I can spend on this. Probably later this weekend, if no one else gets to it.