Grpc-java: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available

Created on 17 Feb 2019  ·  35Comments  ·  Source: grpc/grpc-java

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.13.1

What did you expect to see?

The jar should be running fine.

I am using java 8 to build an executable jar. Below is the java version:

$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-2~14.04-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

I am using gradle 3.4.1 to generate the jar as follows:
/opt/gradle-3.4.1/bin/gradle jar -Dorg.gradle.java.home=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
Below are the dependencies defined in the module-level build.gradle file:

dependencies {
  compile files("$TOOLCHAIN_VERSION_DIR/lib/commons-io-2.6.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/grpc-all.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/java-protobuf.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-core-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-slf4j-impl-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/netty-tcnative-boringssl-static-2.0.20.Final.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/netty-tcnative-2.0.20.Final.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/picocli-3.8.2.jar")
  compile group: 'com.google.guava', name: 'guava', version: '20.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/javassist-3.19.0-GA.jar")
  compile project(':annotation')
}

After building, I am running the jar on an AIX7.2 machine as follows:
/usr/java8_64/jre/bin/java -jar agent-1.0.jar

The java version on the AIX machine is as follows:

$ /usr/java8_64/jre/bin/java -version           
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.26 - pap6480sr5fp26-20181115_03(SR5 FP26))
IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64-Bit Compressed References 20181106_401576 (JIT enabled, AOT enabled)
OpenJ9   - fde1d6f
OMR      - d8c3617
IBM      - 5c4a9f0)
JCL - 20181022_01 based on Oracle jdk8u191-b26

But I am getting the following error while running:

Feb 17, 2019 3:11:33 AM io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: netty-tcnative unavailable (this may be normal)
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_aix_ppc_64, netty_tcnative_ppc_64, netty_tcnative]
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:93)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:430)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:242)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:456)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:55)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:91)
        at com.ankit.agents.AgentMain.main(AgentMain.java:132)
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_aix_ppc_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_aix_ppc_64.a
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_ppc_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_ppc_64.a
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative.a
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                                ... 11 more

Feb 17, 2019 3:11:33 AM io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Conscrypt not found (this may be normal)
Feb 17, 2019 3:11:33 AM io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Jetty ALPN unavailable (this may be normal)
java.lang.ClassNotFoundException: org.eclipse.jetty.alpn.ALPN
        at java.lang.Class.forNameImpl(Native Method)
        at java.lang.Class.forName(Class.java:403)
        at io.grpc.netty.JettyTlsUtil.isJettyAlpnConfigured(JettyTlsUtil.java:64)
        at io.grpc.netty.JettyTlsUtil.getJettyAlpnUnavailabilityCause(JettyTlsUtil.java:75)
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:255)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:456)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:55)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:91)
        at com.ankit.agents.AgentMain.main(AgentMain.java:132)

Exception in thread "main" java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:456)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:55)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:91)
        at com.ankit.agents.AgentMain.main(AgentMain.java:132)

However, it is running fine on an Ubuntu machine. I am running the jar as follows:
/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java -jar agent-1.0.jar

The java version here is:

$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-2~14.04-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

What is the problem here and how to solve this?

picture ankitshubham97

Most helpful comment

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

picture crengutam on 17 May 2019
👍20 🎉5 👀1 ❤1

All 35 comments

You need to have an (exact) version match for the netty and tcnative versions to work together.
See also the table at https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

grpc-version: 1.13.1
tcnative-version: 2.0.20.Final (but should be 2.0.8.Final)

Off-Topic: Could we add a headline/link anchor there so that the table is easier to link?

ST-DDT picture ST-DDT on 18 Feb 2019
👍2

@ST-DDT I tried to use grpc 1.18.0 which matches tcnative-version 2.0.20.Final. But it is giving the same error. My guess is that since I have built it on an ubuntu machine using openjdk, it is creating problem for IBM JRE. By the way, is there any option other than netty-tcnative that can be used in my use case? Are you aware of some? I would appreciate any help.

ankitshubham97 picture ankitshubham97 on 18 Feb 2019
👍1

Usually I just use grpc-netty with netty-tcnative-boringssl-static no need for an extra netty-tcnative.

As an alternative have you tried using grpc-netty-shaded? AFAICT this has the necessary/compatibile grpc/netty/tcnative libraries included.

ST-DDT picture ST-DDT on 18 Feb 2019

@ST-DDT Changed the dependency in the build.gradle to following:

dependencies {
  compile files("$TOOLCHAIN_VERSION_DIR/lib/commons-io-2.6.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/grpc-all-1.18.0.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/java-protobuf.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-core-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-slf4j-impl-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/netty-tcnative-boringssl-static-2.0.20.Final.jar")
  compile group: 'io.grpc', name: 'grpc-netty', version: '1.18.0'
  compile group: 'io.grpc', name: 'grpc-netty-shaded', version: '1.18.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/picocli-3.8.2.jar")
  compile group: 'com.google.guava', name: 'guava', version: '20.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/javassist-3.19.0-GA.jar")
  compile project(':annotation')
}

It is still not working. I think openjdk is not build aix native libraries (as seen in the log).

ankitshubham97 picture ankitshubham97 on 18 Feb 2019

I've been trying to solve the same issue for two weeks, I've tried this also, it happens during configure stackdriver logging configuration in a cluster
RuntimeException in Action for tag [appender] com.google.cloud.logging.LoggingException: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available

khodayar picture khodayar on 18 Feb 2019

Can you paste the logs if you just use the dependencies as described in the readme (+plus the non-grpc/non-netty/non-tcnative libs)?

groovy compile 'io.grpc:grpc-netty-shaded:1.18.0' compile 'io.grpc:grpc-protobuf:1.18.0' compile 'io.grpc:grpc-stub:1.18.0'

ST-DDT picture ST-DDT on 18 Feb 2019
👍1

this is the error log when spring boot app wants to start :

ERROR in ch.qos.logback.core.joran.spi.Interpreter@23:16 - RuntimeException in Action for tag [appender] com.google.cloud.logging.LoggingException: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
        at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:169)
        at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:82)
        at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
        at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:117)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:264)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:237)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:200)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:173)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74)
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:338)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:297) ....

this is the dependencies tree:

 --- maven-dependency-plugin:3.0.2:tree (default-cli) @ bootservice ---
 com.kody.c:boot-service:jar:0.0.1-SNAPSHOT
 +- org.springframework.boot:spring-boot-starter-actuator:jar:2.0.8.RELEASE:compile
 |  +- org.springframework.boot:spring-boot-starter:jar:2.0.8.RELEASE:compile
 |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.8.RELEASE:compile
 |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
 |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
 |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
 |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
 |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
 |  +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:2.0.8.RELEASE:compile
 |  |  +- org.springframework.boot:spring-boot-actuator:jar:2.0.8.RELEASE:compile
 |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.8:compile
 |  \- io.micrometer:micrometer-core:jar:1.0.9:compile
 |     +- org.hdrhistogram:HdrHistogram:jar:2.1.10:compile
 |     \- org.latencyutils:LatencyUtils:jar:2.0.3:compile
 +- org.springframework.boot:spring-boot-starter-web:jar:2.0.8.RELEASE:compile
 |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.8.RELEASE:compile
 |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.8:compile
 |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.8:compile
 |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.8.RELEASE:compile
 |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.37:compile
 |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.37:compile
 |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.37:compile
 |  +- org.hibernate.validator:hibernate-validator:jar:6.0.14.Final:compile
 |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
 |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
 |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
 |  +- org.springframework:spring-web:jar:5.0.12.RELEASE:compile
 |  \- org.springframework:spring-webmvc:jar:5.0.12.RELEASE:compile
 |     \- org.springframework:spring-expression:jar:5.0.12.RELEASE:compile
 +- org.springframework.security.oauth.boot:spring-security-oauth2-autoconfigure:jar:2.0.0.RELEASE:compile
 |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
 |  +- org.springframework.boot:spring-boot:jar:2.0.8.RELEASE:compile
 |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.8.RELEASE:compile
 |  \- org.springframework.security:spring-security-jwt:jar:1.0.9.RELEASE:compile
 |     \- org.bouncycastle:bcpkix-jdk15on:jar:1.56:compile
 +- org.springframework.boot:spring-boot-starter-test:jar:2.0.8.RELEASE:test
 |  +- org.springframework.boot:spring-boot-test:jar:2.0.8.RELEASE:test
 |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.0.8.RELEASE:test
 |  +- com.jayway.jsonpath:json-path:jar:2.4.0:test
 |  |  \- net.minidev:json-smart:jar:2.3:test
 |  |     \- net.minidev:accessors-smart:jar:1.2:test
 |  +- junit:junit:jar:4.12:test
 |  +- org.assertj:assertj-core:jar:3.9.1:test
 |  +- org.mockito:mockito-core:jar:2.15.0:test
 |  |  +- net.bytebuddy:byte-buddy:jar:1.7.11:test
 |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
 |  |  \- org.objenesis:objenesis:jar:2.6:test
 |  +- org.hamcrest:hamcrest-core:jar:1.3:test
 |  +- org.hamcrest:hamcrest-library:jar:1.3:test
 |  +- org.skyscreamer:jsonassert:jar:1.5.0:test
 |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
 |  +- org.springframework:spring-core:jar:5.0.12.RELEASE:compile
 |  |  \- org.springframework:spring-jcl:jar:5.0.12.RELEASE:compile
 |  +- org.springframework:spring-test:jar:5.0.12.RELEASE:test
 |  \- org.xmlunit:xmlunit-core:jar:2.5.1:test
 +- org.springframework.boot:spring-boot-starter-aop:jar:2.0.8.RELEASE:compile
 |  +- org.springframework:spring-aop:jar:5.0.12.RELEASE:compile
 |  \- org.aspectj:aspectjweaver:jar:1.8.13:compile
 +- org.springframework.cloud:spring-cloud-gcp-starter-logging:jar:1.0.0.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-gcp-starter:jar:1.0.0.RELEASE:compile
 |  |  +- org.springframework.cloud:spring-cloud-gcp-core:jar:1.0.0.RELEASE:compile
 |  |  \- org.springframework.cloud:spring-cloud-gcp-autoconfigure:jar:1.0.0.RELEASE:compile
 |  +- ch.qos.logback.contrib:logback-json-classic:jar:0.1.5:compile
 |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
 |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
 |  |  \- ch.qos.logback.contrib:logback-json-core:jar:0.1.5:compile
 |  +- com.google.cloud:google-cloud-logging-logback:jar:0.56.0-alpha:compile
 |  |  \- com.google.cloud:google-cloud-logging:jar:1.38.0:compile
 |  |     +- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.21.0:compile
 |  |     \- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
 |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
 +- io.netty:netty-tcnative-boringssl-static:jar:2.0.20.Final:compile
 +- org.springframework.cloud:spring-cloud-gcp-starter-trace:jar:1.1.0.M3:compile
 |  +- org.springframework.cloud:spring-cloud-starter-sleuth:jar:2.0.1.RELEASE:compile
 |  |  \- org.springframework.cloud:spring-cloud-sleuth-core:jar:2.0.1.RELEASE:compile
 |  |     +- org.aspectj:aspectjrt:jar:1.8.13:compile
 |  |     +- io.zipkin.brave:brave-context-log4j2:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-spring-web:jar:5.1.4:compile
 |  |     |  \- io.zipkin.brave:brave-instrumentation-http:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-spring-rabbit:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-kafka-clients:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-httpclient:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-httpasyncclient:jar:5.1.4:compile
 |  |     \- io.zipkin.brave:brave-instrumentation-spring-webmvc:jar:5.1.4:compile
 |  |        \- io.zipkin.brave:brave-instrumentation-servlet:jar:5.1.4:compile
 |  +- io.zipkin.gcp:zipkin-sender-stackdriver:jar:0.8.3:compile
 |  |  +- io.zipkin.gcp:zipkin-translation-stackdriver:jar:0.8.3:compile
 |  |  +- io.zipkin.reporter2:zipkin-reporter:jar:2.7.10:compile
 |  |  +- com.google.api.grpc:grpc-google-devtools-cloudtrace-v1:jar:0.1.1:compile
 |  |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.1:compile
 |  |  +- io.grpc:grpc-core:jar:1.18.0:compile
 |  |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
 |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.2.0:compile
 |  |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
 |  |  |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
 |  |  |  +- org.checkerframework:checker-compat-qual:jar:2.5.2:compile
 |  |  |  +- io.opencensus:opencensus-api:jar:0.18.0:compile
 |  |  |  \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.18.0:compile
 |  |  +- io.grpc:grpc-protobuf:jar:1.18.0:compile
 |  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.18.0:compile
 |  |  +- io.zipkin.zipkin2:zipkin:test-jar:tests:2.11.7:compile
 |  |  \- io.zipkin.zipkin2:zipkin:jar:2.11.7:compile
 |  +- io.zipkin.gcp:brave-propagation-stackdriver:jar:0.8.3:compile
 |  |  \- io.zipkin.brave:brave:jar:5.4.3:compile
 |  +- org.springframework.cloud:spring-cloud-sleuth-zipkin:jar:2.0.1.RELEASE:compile
 |  |  +- org.springframework.cloud:spring-cloud-commons:jar:2.0.1.RELEASE:compile
 |  |  |  \- org.springframework.security:spring-security-crypto:jar:5.0.11.RELEASE:compile
 |  |  +- io.zipkin.reporter2:zipkin-sender-kafka11:jar:2.7.7:compile
 |  |  \- io.zipkin.reporter2:zipkin-sender-amqp-client:jar:2.7.7:compile
 |  \- com.google.cloud:google-cloud-core-grpc:jar:1.53.0:compile
 |     +- com.google.auth:google-auth-library-credentials:jar:0.12.0:compile
 |     +- com.google.cloud:google-cloud-core:jar:1.53.0:compile
 |     |  +- joda-time:joda-time:jar:2.9.9:compile
 |     |  +- com.google.http-client:google-http-client:jar:1.27.0:compile
 |     |  +- com.google.api:api-common:jar:1.7.0:compile
 |     |  +- com.google.api:gax:jar:1.35.0:compile
 |     |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.12.0:compile
 |     +- com.google.guava:guava:jar:26.0-android:compile
 |     |  \- com.google.j2objc:j2objc-annotations:jar:1.1:compile
 |     +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
 |     +- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile
 |     +- io.grpc:grpc-context:jar:1.16.1:compile
 |     +- io.grpc:grpc-netty-shaded:jar:1.18.0:compile
 |     +- io.grpc:grpc-stub:jar:1.18.0:compile
 |     +- io.grpc:grpc-auth:jar:1.16.1:compile
 |     \- com.google.api:gax-grpc:jar:1.35.0:compile
 |        +- org.threeten:threetenbp:jar:1.3.3:compile
 |        \- com.google.auth:google-auth-library-oauth2-http:jar:0.11.0:compile
 |           \- com.google.http-client:google-http-client-jackson2:jar:1.24.1:compile
 +- org.springframework.cloud:spring-cloud-starter-netflix-hystrix:jar:2.0.1.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-starter:jar:2.0.1.RELEASE:compile
 |  |  +- org.springframework.cloud:spring-cloud-context:jar:2.0.1.RELEASE:compile
 |  |  \- org.springframework.security:spring-security-rsa:jar:1.0.5.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-netflix-core:jar:2.0.1.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-netflix-ribbon:jar:2.0.1.RELEASE:compile
 |  |  \- org.springframework.cloud:spring-cloud-netflix-archaius:jar:2.0.1.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar:2.0.1.RELEASE:compile
 |  |  +- com.netflix.archaius:archaius-core:jar:0.7.6:compile
 |  |  \- commons-configuration:commons-configuration:jar:1.8:compile
 |  |     \- commons-lang:commons-lang:jar:2.6:compile
 |  +- com.netflix.hystrix:hystrix-core:jar:1.5.12:compile
 |  |  \- io.reactivex:rxjava:jar:1.3.8:compile
 |  +- com.netflix.hystrix:hystrix-serialization:jar:1.5.12:compile
 |  |  \- com.fasterxml.jackson.module:jackson-module-afterburner:jar:2.9.8:runtime
 |  +- com.netflix.hystrix:hystrix-metrics-event-stream:jar:1.5.12:compile
 |  +- com.netflix.hystrix:hystrix-javanica:jar:1.5.12:compile
 |  |  +- org.apache.commons:commons-lang3:jar:3.7:runtime
 |  |  \- org.ow2.asm:asm:jar:5.0.4:runtime
 |  \- io.reactivex:rxjava-reactive-streams:jar:1.2.1:compile
 |     \- org.reactivestreams:reactive-streams:jar:1.0.2:compile
 +- org.springframework.cloud:spring-cloud-starter-contract-stub-runner:jar:2.0.1.RELEASE:test
 |  +- org.springframework.cloud:spring-cloud-contract-stub-runner:jar:2.0.1.RELEASE:test
 |  |  +- org.springframework.cloud:spring-cloud-contract-shade:jar:2.0.1.RELEASE:test
 |  |  +- org.springframework.cloud:spring-cloud-contract-spec:jar:2.0.1.RELEASE:test
 |  |  |  +- dk.brics.automaton:automaton:jar:1.11-8:test
 |  |  |  \- org.apache.commons:commons-text:jar:1.1:test
 |  |  +- org.eclipse.jgit:org.eclipse.jgit:jar:4.11.0.201803080745-r:test
 |  |  |  +- com.jcraft:jsch:jar:0.1.54:test
 |  |  |  +- com.jcraft:jzlib:jar:1.1.1:test
 |  |  |  \- com.googlecode.javaewah:JavaEWAH:jar:1.1.6:test
 |  |  +- com.jcraft:jsch.agentproxy.sshagent:jar:0.0.9:test
 |  |  |  \- com.jcraft:jsch.agentproxy.core:jar:0.0.9:test
 |  |  +- com.jcraft:jsch.agentproxy.jsch:jar:0.0.9:test
 |  |  \- com.jcraft:jsch.agentproxy.usocket-jna:jar:0.0.9:test
 |  |     +- net.java.dev.jna:jna:jar:4.5.2:test
 |  |     \- net.java.dev.jna:jna-platform:jar:4.5.2:test
 |  +- org.springframework.cloud:spring-cloud-stream-test-support:jar:2.0.1.RELEASE:test
 |  |  \- org.springframework.cloud:spring-cloud-stream:jar:2.0.1.RELEASE:test
 |  |     +- org.springframework.boot:spring-boot-starter-validation:jar:2.0.8.RELEASE:test
 |  |     +- org.springframework:spring-messaging:jar:5.0.12.RELEASE:test
 |  |     +- org.springframework.integration:spring-integration-core:jar:5.0.11.RELEASE:test
 |  |     |  +- org.springframework:spring-tx:jar:5.0.12.RELEASE:test
 |  |     |  \- io.projectreactor:reactor-core:jar:3.1.14.RELEASE:compile
 |  |     +- org.springframework.integration:spring-integration-jmx:jar:5.0.11.RELEASE:test
 |  |     +- org.springframework:spring-tuple:jar:1.0.0.RELEASE:test
 |  |     |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:test
 |  |     |     \- com.esotericsoftware:minlog:jar:1.3.0:test
 |  |     +- org.springframework.integration:spring-integration-tuple:jar:1.0.0.RELEASE:test
 |  |     \- org.springframework.retry:spring-retry:jar:1.2.3.RELEASE:test
 |  +- org.springframework.cloud:spring-cloud-contract-wiremock:jar:2.0.1.RELEASE:test
 |  |  +- com.github.tomakehurst:wiremock-standalone:jar:2.18.0:test
 |  |  \- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
 |  |     \- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
 |  \- org.eclipse.aether:aether-api:jar:1.0.2.v20150114:test
 +- org.springframework.cloud:spring-cloud-starter-contract-verifier:jar:2.0.1.RELEASE:test
 |  +- org.springframework.cloud:spring-cloud-contract-verifier:jar:2.0.1.RELEASE:test
 |  |  +- javax.inject:javax.inject:jar:1:test
 |  |  +- org.codehaus.groovy:groovy:jar:2.4.16:test
 |  |  +- org.codehaus.groovy:groovy-nio:jar:2.4.16:test
 |  |  +- org.codehaus.groovy:groovy-json:jar:2.4.16:test
 |  |  +- org.codehaus.groovy:groovy-xml:jar:2.4.16:test
 |  |  +- com.github.jknack:handlebars:jar:4.0.6:test
 |  |  |  +- org.antlr:antlr4-runtime:jar:4.5.1-1:test
 |  |  |  \- org.mozilla:rhino:jar:1.7R4:test
 |  |  \- commons-beanutils:commons-beanutils:jar:1.9.3:test
 |  |     +- commons-logging:commons-logging:jar:1.2:test
 |  |     \- commons-collections:commons-collections:jar:3.2.2:test
 |  +- io.rest-assured:rest-assured:jar:3.0.7:test
 |  |  +- org.apache.httpcomponents:httpmime:jar:4.5.6:test
 |  |  +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:test
 |  |  +- io.rest-assured:json-path:jar:3.0.7:test
 |  |  |  \- io.rest-assured:rest-assured-common:jar:3.0.7:test
 |  |  \- io.rest-assured:xml-path:jar:3.0.7:test
 |  |     \- javax.xml.bind:jaxb-api:jar:2.3.1:test
 |  |        \- javax.activation:javax.activation-api:jar:1.2.0:test
 |  +- io.rest-assured:spring-mock-mvc:jar:3.0.7:test
 |  +- com.toomuchcoding.jsonassert:jsonassert:jar:0.4.13:test
 |  \- cglib:cglib:jar:3.2.4:test
 |     \- org.apache.ant:ant:jar:1.9.6:test
 |        \- org.apache.ant:ant-launcher:jar:1.9.6:test
 +- com.googlecode.junit-toolbox:junit-toolbox:jar:1.5:test
 |  +- commons-io:commons-io:jar:2.4:test
 |  \- org.codehaus.jsr166-mirror:jsr166y:jar:1.7.0:test
 +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.8:compile
 +- com.fasterxml.jackson.core:jackson-core:jar:2.9.8:compile
 +- commons-codec:commons-codec:jar:1.9:compile
 +- org.bouncycastle:bcprov-jdk15on:jar:1.60:compile
 \- org.springframework.security.oauth:spring-security-oauth2:jar:2.1.3.RELEASE:compile
    +- org.springframework:spring-beans:jar:5.0.12.RELEASE:compile
    +- org.springframework:spring-context:jar:5.0.12.RELEASE:compile
    +- org.springframework.security:spring-security-core:jar:5.0.11.RELEASE:compile
    +- org.springframework.security:spring-security-config:jar:5.0.11.RELEASE:compile
    +- org.springframework.security:spring-security-web:jar:5.0.11.RELEASE:compile
    \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
       \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
khodayar picture khodayar on 18 Feb 2019

@khodayar Are you trying to run this on AIX too?

ankitshubham97 picture ankitshubham97 on 18 Feb 2019

@ankitshubham97
I'm trying to run it on a local kubernetes cluster or gcp, I can run the code locally without any problem, but not in the clusters

khodayar picture khodayar on 18 Feb 2019

@khodayar What is the underlying OS?

ankitshubham97 picture ankitshubham97 on 18 Feb 2019

I'm using docker for windows but Linux containers

khodayar picture khodayar on 18 Feb 2019

Is that the complete stacktrace?
Where is the grpc stuff?
Also you use different versions for the grpc components try updating them to the same version?

ST-DDT picture ST-DDT on 19 Feb 2019

Here is the complete stacktrace:
bash-4.4# /usr/java8_64/jre/bin/java -Xbootclasspath/p:/home/ankit/alpn-boot-8.1.13.v20180117.jar -jar agent-1.0.jar

Feb 19, 2019 12:02:34 AM io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: netty-tcnative unavailable (this may be normal)
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_aix_ppc_64, netty_tcnative_ppc_64, netty_tcnative]
        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104)
        at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:526)
        at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:93)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:244)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:471)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:53)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:52)
        at com.ankit.agents.AgentMain.main(AgentMain.java:93)
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative_aix_ppc_64
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative_aix_ppc_64.a
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative_ppc_64
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative_ppc_64.a
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative.a
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 11 more

Feb 19, 2019 12:02:34 AM io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Conscrypt not found (this may be normal)
Feb 19, 2019 12:02:34 AM io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Jetty ALPN unavailable (this may be normal)
java.lang.ClassNotFoundException: org.eclipse.jetty.alpn.ALPN
        at java.lang.Class.forNameImpl(Native Method)
        at java.lang.Class.forName(Class.java:403)
        at io.grpc.netty.shaded.io.grpc.netty.JettyTlsUtil.isJettyAlpnConfigured(JettyTlsUtil.java:64)
        at io.grpc.netty.shaded.io.grpc.netty.JettyTlsUtil.getJettyAlpnUnavailabilityCause(JettyTlsUtil.java:75)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:257)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:471)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:53)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:52)
        at com.ankit.agents.AgentMain.main(AgentMain.java:93)

Exception in thread "main" java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:258)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:471)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:53)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:52)
        at com.ankit.agents.AgentMain.main(AgentMain.java:93)

My dependency in the build.gradle is like this now:

dependencies {
  compile files("$TOOLCHAIN_VERSION_DIR/lib/commons-io-2.6.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/alpn-api-1.0.0.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/grpc-all-1.18.0.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/java-protobuf.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty-alpn-openjdk8-server-9.4.14.v20181114.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-core-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-slf4j-impl-2.8.jar")
  compile 'io.grpc:grpc-netty-shaded:1.18.0'
  compile 'io.grpc:grpc-protobuf:1.18.0'
  compile 'io.grpc:grpc-stub:1.18.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/picocli-3.8.2.jar")
  compile group: 'com.google.guava', name: 'guava', version: '20.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/javassist-3.19.0-GA.jar")
  compile project(':annotation')
}
ankitshubham97 picture ankitshubham97 on 19 Feb 2019

It is not able to find some native methods("could not load a native library" error in the above log), maybe we have to compile them, but I am not sure how to do that.

ankitshubham97 picture ankitshubham97 on 19 Feb 2019

Is that the complete stacktrace?
Where is the grpc stuff?
Also you use different versions for the grpc components try updating them to the same version?

I used the same versions (1.18.0) for all grpc dependencies , didn't help

khodayar picture khodayar on 19 Feb 2019

One quick thing: you probably don't want netty tcnative static. Most people should not use that. The regular, dynamic version should be correct.

carl-mastrangelo picture carl-mastrangelo on 19 Feb 2019

I added that based on : https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty
Also I had tried the regular version before

khodayar picture khodayar on 19 Feb 2019

@ankitshubham97 /@khodayar
Hey, Did you find a solution for this issue? If yes, Could you please update here. I am also facing the same issue.

rahulKQL picture rahulKQL on 11 Apr 2019

@rahulKQL we couldn't find a solution.

khodayar picture khodayar on 11 Apr 2019

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

crengutam picture crengutam on 17 May 2019
👍20 🎉5 👀1 ❤1

Hey Guys,

Tying to identifying this issues from last 4 days but found the perfect solution that is just checked when

Experiment 1. :- Java 8 32bit with eclipse 32 bit :---- Failed to load any of the given libraries: [netty_tcnative_windows_x86_32, netty_tcnative_x86_32, netty_tcnative]
Experiment 2.:- ### Java 8 64bit with Eclipse Luna 64 bit 👍 :- Worked Without Error.

Thanks

Ravipatel401 picture Ravipatel401 on 8 Jun 2019
👍1

@Ravipatel401 Do you know why it failed to load? It may be something as simple as the filename of the dll was wrong.

carl-mastrangelo picture carl-mastrangelo on 12 Jun 2019

I don't know if this is related, but I noticed that the .so files provided by io.netty:netty-tcnative-boringssl-static:2.0.7.Final and 2.0.12.Final were compiled against a specific set of libraries that expect some symbols to exist in the system's libraries. In my case, I'm running my application in a Docker image based on alpine 3.9.4. The library that I believe grpc is loading (but please correct me if I'm wrong) expects pthread_yield, among others:

bash-4.4# ldd libnetty_tcnative_linux_x86_64.so
    ldd (0x7f3412b6f000)
    librt.so.1 => ldd (0x7f3412b6f000)
    libc.so => ldd (0x7f3412b6f000)
    libpthread.so.0 => ldd (0x7f3412b6f000)
    libdl.so.2 => ldd (0x7f3412b6f000)
Error relocating libnetty_tcnative_linux_x86_64.so: __isnan: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: __isinf: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: __strndup: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: pthread_mutex_consistent_np: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: pthread_mutexattr_setrobust_np: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: pthread_yield: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: sys_siglist: symbol not found

The version of libpthread.so.0 that ships with this version of alpine does not provide these symbols (I haven't investigated where sys_syglist, or the __* functions come from). Is it possible this is the cause of the "could not load native" errors? If so, is it possible for grpc-java to use some other pure-Java TLS provider instead?

dpkirchner picture dpkirchner on 14 Jun 2019

cc @normanmaurer

carl-mastrangelo picture carl-mastrangelo on 14 Jun 2019

@Ravipatel401 Do you know why it failed to load? It may be something as simple as the filename of the dll was wrong.

@carl-mastrangelo Issue is general while you Run program you need to JAVA 64bit (jvm,jre,jdk all should be 64bit ) otherwise it won't run. Make sure you using 64bit OS on 64bit JAVA. and don't try to change and preloaded libraries it's will create more problems.

Ravipatel401 picture Ravipatel401 on 17 Jun 2019

How to make it work in a 32-bit machine then? Can anyone help with that? @ravipatel410

akhil-yalagonda picture akhil-yalagonda on 18 Jun 2019

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Thanks, It's working for me.

rahulgoti1 picture rahulgoti1 on 23 Jul 2019

I still faced the issue while using openjdk:8 and managed to fix it by upgrading to openjdk:11 😊
Java 11 has ALPN available by default.

MrStobbart picture MrStobbart on 31 Jul 2019

It must be 64 bit

On Wed, Jul 31, 2019, 2:20 PM Marek Meyer notifications@github.com wrote:

I still faced the issue while using openjdk:8 and managed to fix it by
upgrading to openjdk:11 😊
Java 11 has ALPN available by default.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-java/issues/5369?email_source=notifications&email_token=AA2RM3I2PS5XT6TSC7X2Q3LQCFG5ZA5CNFSM4GX6BP3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3GR2LI#issuecomment-516758829,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA2RM3KZDF2LCOD6JUVNDILQCFG5ZANCNFSM4GX6BP3A
.

Ravipatel401 picture Ravipatel401 on 31 Jul 2019
👍2

I had the same issue and I resolved changing the java version to 64 bit!

mrojasabregu picture mrojasabregu on 11 Dec 2019
👍1

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Thanks, It's working for me.

Yes, this worked for me too.. Worked like magic!!!! Thanks for sharing :)

damioti picture damioti on 13 Jan 2020

Welcome but ensure for future if you are anyof use netty libraries always
makeup with 64bit jdk jre and all with java above 8 version...

On Tue, 14 Jan 2020 at 12:30 AM, damioti notifications@github.com wrote:

I had the same problem and I was using a Docker image with a kernel that
was incompatible with the netty native libraries needed by my app. I
changed the image in my Dockerfile(in my case from "FROM
openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Thanks, It's working for me.

Yes, this worked for me too.. Worked like magic!!!! Thanks for sharing :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-java/issues/5369?email_source=notifications&email_token=AA2RM3LKLYYYBVZJGIYJLHDQ5S26VA5CNFSM4GX6BP3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIZ4I3Y#issuecomment-573817967,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2RM3IRJKVH7SP4EFL5SDLQ5S26VANCNFSM4GX6BP3A
.

Ravipatel401 picture Ravipatel401 on 13 Jan 2020

As @Ravipatel401 already said netty does not run on 32bit.
Another factor here is that netty is not running on alpine images in general, see https://github.com/netty/netty-tcnative/issues/111.

appleseedexm picture appleseedexm on 14 Feb 2020

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Gracias !! Fue mi solucion luego de no poder dormir por ese problema

liopalacios picture liopalacios on 14 Jun 2020

Upgrading to Java 11 was a long-standing workaround for this problem. But since Java 8u252 (April 2020) Java 8 has native support for ALPN, so netty_tcnative isn't required (although still recommended when available) even on Java 8.

ejona86 picture ejona86 on 28 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lbensaad picture lbensaad  ·  7Comments
zhaoweizhen picture zhaoweizhen  ·  8Comments
emaayan picture emaayan  ·  8Comments
chsi13 picture chsi13  ·  7Comments
lamaxiyali picture lamaxiyali  ·  4Comments