Google-cloud-java: PubSub grpc configuration error

[Mistic92]

Hello. I'm using PubSub 0.20.0-beta but I have this problem since few versions.
I'm using it with spring boot with Tomcat and deploying to kubernetes on compute engine with COS image.

When trying to push message I get error "Jetty ALPN/NPN has not been properly configured".
I have tried to use Jetty but I got the same error. What's weird, locally it's ok and everything is working.

I saw there is #2039 where upgrade of grpc lib is mentioned but there is no separate issue for this error.

2017-06-24T16:20:34.20659867Z java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
2017-06-24T16:20:34.206602706Z  at io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.20660692Z   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206610273Z  at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206613683Z  at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206617344Z  at io.grpc.netty.NettyChannelBuilder.createProtocolNegotiatorByType(NettyChannelBuilder.java:335) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206620776Z  at io.grpc.netty.NettyChannelBuilder.createProtocolNegotiator(NettyChannelBuilder.java:308) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206624331Z  at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DynamicNettyTransportParams.getProtocolNegotiator(NettyChannelBuilder.java:499) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206627826Z  at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.newClientTransport(NettyChannelBuilder.java:448) ~[grpc-netty-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206632384Z  at io.grpc.internal.CallCredentialsApplyingTransportFactory.newClientTransport(CallCredentialsApplyingTransportFactory.java:61) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.20663599Z   at io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:209) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206639526Z  at io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:186) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206658614Z  at io.grpc.internal.ManagedChannelImpl$SubchannelImplImpl.obtainActiveTransport(ManagedChannelImpl.java:806) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.20666273Z   at io.grpc.internal.GrpcUtil.getTransportFromPickResult(GrpcUtil.java:568) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206666037Z  at io.grpc.internal.DelayedClientTransport.reprocess(DelayedClientTransport.java:296) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206669031Z  at io.grpc.internal.ManagedChannelImpl$LbHelperImpl$5.run(ManagedChannelImpl.java:724) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206672086Z  at io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:87) ~[grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206675044Z  at io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:715) [grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206678244Z  at io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onUpdate(ManagedChannelImpl.java:752) [grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.20668125Z   at io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:174) [grpc-core-1.2.0.jar!/:1.2.0]
2017-06-24T16:20:34.206684289Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111-internal]
2017-06-24T16:20:34.206687236Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111-internal]
2017-06-24T16:20:34.206690707Z  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-internal]

[shinfan]

@Mistic92 Hava you tried to use the following config which forces spring to use Jetty?

 <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
      <version>1.5.3.RELEASE</version>
      <exclusions>
        <exclusion>
          <!-- Use Jetty instead of Tomcat
               https://www.mkyong.com/spring-boot/spring-boot-jetty-as-embedded-server/ -->
          <groupId>org.springframework.boot</groupId>
          <artifactId>spring-boot-starter-tomcat</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-jetty</artifactId>
      <version>1.5.3.RELEASE</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.security</groupId>
      <artifactId>spring-security-core</artifactId>
      <version>4.2.2.RELEASE</version>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-pubsub</artifactId>
      <version>0.17.2-alpha</version>
    </dependency>

[Mistic92]

I see #2171 is almost merged so I'll wait and check if it's working now ;)
And yes, with gradle I'm using config which excluse tomcat from starter.

[Mistic92]

Ok, something is not working :( Now it's crashing during creating beans. Again only after deployment to kubernetes.

My dependencies

dependencies {
    compile 'org.slf4j:slf4j-api:1.7.24'
    compile group: 'org.projectlombok', name: 'lombok', version: '1.16.14'

    compile("org.springframework.boot:spring-boot-starter-web:1.5.3.RELEASE")
    compile("org.springframework.boot:spring-boot-starter-actuator")
    compile("org.springframework.boot:spring-boot-starter-security:1.5.3.RELEASE")
    compile("org.springframework.boot:spring-boot-starter-logging")
    compile("org.springframework.boot:spring-boot-starter-jetty:1.5.3.RELEASE")

    compile("org.springframework.social:spring-social-facebook:3.0.0.M3")
    compile("com.github.spring-social:spring-social-google:latest.release")
    compile("org.springframework.social:spring-social-core")


    compile('com.google.cloud:google-cloud-datastore:1.2.0') {
    }
    compile('com.google.cloud:google-cloud-pubsub:0.20.1-beta') {
    }

    compile 'com.github.sai-pullabhotla:catatumbo:catatumbo-2.1.0'
//    compile 'com.github.Mistic92:catatumbo:-SNAPSHOT'
    compile('com.google.guava:guava:20.0') { force = true }

    //MIT
    compile group: 'de.bytefish.fcmjava', name: 'fcmjava-core', version: '1.1'
    compile group: 'de.bytefish.fcmjava', name: 'fcmjava-parent', version: '1.1'
    compile group: 'de.bytefish.fcmjava', name: 'fcmjava-client', version: '1.1'

    compile group: 'javax.validation', name: 'validation-api', version: '2.0.0.Beta1'

    compile group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.8.7'

    compile 'com.google.firebase:firebase-admin:5.2.0'

    compile "io.reactivex.rxjava2:rxjava:2.1.0"

    testCompile 'junit:junit:4.12'
}

configurations {
    compile.exclude module: "spring-boot-starter-tomcat"
//    all*.exclude group: 'com.google.guava', module: 'guava', version: '19.0'
//    all*.exclude group: 'com.google.guava', module: 'guava', version: '18.0'

}

Log from kubernetes

2017-06-27T13:51:07.401271705Z Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.google.cloud.pubsub.v1.TopicAdminClient]: Factory method 'provideTopicAdminClient' threw exception; nested exception is java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
2017-06-27T13:51:07.40127596Z   at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
2017-06-27T13:51:07.401279317Z  at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
2017-06-27T13:51:07.401282534Z  ... 39 common frames omitted
2017-06-27T13:51:07.401285477Z Caused by: java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
2017-06-27T13:51:07.401288622Z  at io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174) ~[grpc-netty-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.401291551Z  at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151) ~[grpc-netty-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.401294682Z  at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139) ~[grpc-netty-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.401297747Z  at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109) ~[grpc-netty-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.401302502Z  at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:470) ~[grpc-netty-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.401306025Z  at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:338) ~[grpc-netty-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.40130917Z   at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:305) ~[grpc-core-1.4.0.jar!/:1.4.0]
2017-06-27T13:51:07.401312351Z  at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:139) ~[gax-grpc-0.21.1.jar!/:na]
2017-06-27T13:51:07.401315567Z  at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:116) ~[gax-grpc-0.21.1.jar!/:na]
2017-06-27T13:51:07.401318511Z  at com.google.api.gax.grpc.ChannelAndExecutor.create(ChannelAndExecutor.java:65) ~[gax-grpc-0.21.1.jar!/:na]
2017-06-27T13:51:07.401321517Z  at com.google.api.gax.grpc.ClientSettings.getChannelAndExecutor(ClientSettings.java:91) ~[gax-grpc-0.21.1.jar!/:na]
2017-06-27T13:51:07.401324547Z  at com.google.cloud.pubsub.v1.TopicAdminClient.<init>(TopicAdminClient.java:150) ~[google-cloud-pubsub-0.20.1-beta.jar!/:0.20.1-beta]
2017-06-27T13:51:07.401327864Z  at com.google.cloud.pubsub.v1.TopicAdminClient.create(TopicAdminClient.java:141) ~[google-cloud-pubsub-0.20.1-beta.jar!/:0.20.1-beta]
2017-06-27T13:51:07.401330921Z  at com.google.cloud.pubsub.v1.TopicAdminClient.create(TopicAdminClient.java:133) ~[google-cloud-pubsub-0.20.1-beta.jar!/:0.20.1-beta]
2017-06-27T13:51:07.401334025Z  at com.blenge.blenge.challenge.PubSubChallengeConfig.provideTopicAdminClient(PubSubChallengeConfig.java:35) ~[classes!/:na]
2017-06-27T13:51:07.401337285Z  at com.blenge.blenge.challenge.PubSubChallengeConfig$$EnhancerBySpringCGLIB$$28bcd485.CGLIB$provideTopicAdminClient$1(<generated>) ~[classes!/:na]
2017-06-27T13:51:07.401340744Z  at com.blenge.blenge.challenge.PubSubChallengeConfig$$EnhancerBySpringCGLIB$$28bcd485$$FastClassBySpringCGLIB$$7b294fc3.invoke(<generated>) ~[classes!/:na]
2017-06-27T13:51:07.401344058Z  at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
2017-06-27T13:51:07.40134732Z   at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358) ~[spring-context-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
2017-06-27T13:51:07.401350387Z  at com.blenge.blenge.challenge.PubSubChallengeConfig$$EnhancerBySpringCGLIB$$28bcd485.provideTopicAdminClient(<generated>) ~[classes!/:na]
2017-06-27T13:51:07.401353491Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111-internal]
2017-06-27T13:51:07.401356407Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111-internal]
2017-06-27T13:51:07.401359272Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111-internal]
2017-06-27T13:51:07.401364058Z  at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111-internal]
2017-06-27T13:51:07.401368955Z  at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) ~[spring-beans-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
2017-06-27T13:51:07.401372204Z  ... 40 common frames omitted

[Mistic92]

I found problem and solution.
For docker I'm using openjdk:8u131-jre-alpine and to make grpc working you need to install libc6-compat

FROM openjdk:8u131-jre-alpine
...
RUN apk add --no-cache libc6-compat
...

[pclair]

I am running into same issue on mac, spring boot with jetty container. is there a solution ?