Pulsar: Not able to start Pulsar in Standalone mode with Functions-Worker

Created on 17 Jun 2020 · 18Comments · Source: apache/pulsar

Describe the bug
I am trying to run a pulsar producer using java client that writes data to the standalone pulsar. I have run the pulsar using the following command:

bin/pulsar standalone

When I was writing the data with TLS encryption, it was working fine. But after I tried the TLS authentication configuration, I am getting the following error in my client logs:

09:08:35.261 [AsyncHttpClient-80-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/functions/assignments] Failed to perform http put request: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
09:08:35.273 [main] ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker
org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: HTTP 401 Unauthorized
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:212) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_242]
Caused by: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
    at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1080) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:99) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    ... 47 more

However, when I tried with the following command the standalone pulsar works:

bin/pulsar standalone -nfw -nss

But why is it not working with functions worker ?

To Reproduce
I have followed the following link for creating tls certificates : http://pulsar.apache.org/docs/en/security-tls-transport/

standalone.conf:

tlsEnabled=true
tlsCertRefreshCheckDurationSec=300
tlsCertificateFilePath=/home/Downloads/my-ca/broker.cert.pem
tlsKeyFilePath=/home/Downloads/my-ca/broker.key-pk8.pem
tlsTrustCertsFilePath=/home/Downloads/my-ca/certs/ca.cert.pem
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
authorizationEnabled=false
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
authorizationAllowWildcardsMatching=false
superUserRoles=admin
brokerClientTlsEnabled=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters=tlsCertFile:/home/Downloads/my-ca/admin.cert.pem,tlsKeyFile:/home/Downloads/my-ca/admin.key-pk8.pem
brokerClientTrustCertsFilePath=/home/Downloads/my-ca/certs/ca.cert.pem

client.conf:

webServiceUrl=https://localhost:8443/
brokerServiceUrl=pulsar+ssl://localhost:6651/
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
authParams=tlsCertFile:/home/Downloads/my-ca/abc.cert.pem,tlsKeyFile:/home/Downloads/my-ca/abc.key-pk8.pem
tlsTrustCertsFilePath=/home/Downloads/my-ca/certs/ca.cert.pem

functions-worker.yml:

authenticationEnabled: true
authorizationEnabled: true
tlsEnabled: true

Expected behaviour
Should have run successfully in standalone mode.

Desktop (please complete the following information):
I am doing everything within an Ubuntu 18.04.4 LTS VM with Pulsar 2.5.0

componensecurity triagweek-25 typbug

Source

97arushisharma

All 18 comments

@97arushisharma

You can try a few things:

Can you do the following things to verify if the admin cert has the super-user permissions?

Disable function worker: bin/pulsar standalone -nfw -nss.
Configure client.conf to use the admin cert: tlsCertFile:/home/Downloads/my-ca/admin.cert.pem,tlsKeyFile:/home/Downloads/my-ca/admin.key-pk8.pem.
Run bin/pulsar-admin clusters list to see if you can list the clusters.

sijie on 18 Jun 2020

Thanks @sijie for the reply. I had followed the above steps, yet I am getting the following error:

$ bin/pulsar-admin clusters list
null
Reason: javax.ws.rs.ProcessingException: Connection refused: localhost/127.0.0.1:8443

I am getting this same behavious for all pulsar-admin commands.

97arushisharma on 18 Jun 2020

Did you configure standalone to enable tls port?

sijie on 18 Jun 2020

Yup. I have added these in standalone.conf:

brokerServicePort=6650
webServicePort=8080
brokerServicePortTls=6651

97arushisharma on 18 Jun 2020

Did you configure 8443?

sijie on 18 Jun 2020

I did not have that. Thanks. So after adding it the pulsar-admin commands work. Here is the output:

$ bin/pulsar-admin clusters list
"standalone"

But I still have this issue as to why I have to use -nfw with the standalone command to run it?Why am I not able to run it without -nfw?

97arushisharma on 18 Jun 2020

@97arushisharma okay. then can you try to start the standalone now?

sijie on 18 Jun 2020

I am able to run pulsar in standalone with -nfw. But, it still does not run without this option.

97arushisharma on 18 Jun 2020

@97arushisharma can you add admin to superUserRoles in function_worker.yml?

sijie on 18 Jun 2020

After adding the superUserRoles: admin, I am getting the following error log on running the bin/pulsar standalone command:

05:23:59.016 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.HashSet<java.lang.Object>` out of VALUE_STRING token
 at [Source: (File); line: 176, column: 17] (through reference chain: org.apache.pulsar.functions.worker.WorkerConfig["superUserRoles"])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1442) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1216) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1126) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.handleNonArray(StringCollectionDeserializer.java:274) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:183) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:173) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:21) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4202) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070) ~[com.fasterxml.jackson.core-jackson-databind-2.10.1.jar:2.10.1]
    at org.apache.pulsar.functions.worker.WorkerConfig.load(WorkerConfig.java:404) ~[org.apache.pulsar-pulsar-functions-runtime-2.5.0.jar:2.5.0]
    at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:273) ~[org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
    at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:119) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]

97arushisharma on 18 Jun 2020

@97arushisharma

superUserRoles:
   - admin

Can you try this?

sijie on 18 Jun 2020

I tried this. This time another Authorization error occurred:

05:34:52.025 [AsyncHttpClient-80-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/functions/assignments] Failed to perform http put request: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
05:34:52.036 [main] ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker
org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: Don't have permission to administrate resources on this tenant
    at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:212) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:130) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
    at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234) ~[org.apache.pulsar-pulsar-client-admin-original-2.5.0.jar:2.5.0]
    at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) ~[io.netty-netty-codec-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_242]
Caused by: javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
    at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1080) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:99) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
    ... 47 more

97arushisharma on 18 Jun 2020

Are you using the admin cert? If you are using the admin cert, it should have the super-user. Are you using the wrong cert?

sijie on 18 Jun 2020

I tried recreating the admin and broker certificate with respective common-name as admin and localhost, just to be sure I am not doing anything wrong. I also added all the TLS AUTH related properties in functions_worker.yml, but I am still getting the above error.

authenticationEnabled: true
authorizationEnabled: true
authenticationProviders:
    - "org.apache.pulsar.broker.authentication.AuthenticationProviderTls"
authorizationProvider: "org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider"
superUserRoles:
    - admin
tlsEnabled: true
tlsCertificateFilePath: "/home/sqlstream/Downloads/my-ca/admin.cert.pem"
tlsKeyFilePath: "/home/sqlstream/Downloads/my-ca/admin.key-pk8.pem" 
tlsTrustCertsFilePath: "/home/sqlstream/Downloads/my-ca/certs/ca.cert.pem"

97arushisharma on 18 Jun 2020

Hello, I have encountered the same problem and solved it through the following configuration

check whether the certificate is correct

openssl x509 -in admin.cert.pem -noout -text
openssl x509 -in broker.cert.pem -noout -text

Make sure that these two certificates do not use the same common name, otherwise, errors may occur

Add configuration useTls=true to conf/functions_worker.yml file in standalone environment.

This is an example of functions_worker.yml in a standalone environment. https://github.com/streamnative/pulsar-beat-output/blob/master/tlsConfig/functions_worker.yml

@97arushisharma

tuteng on 27 Jun 2020

👍1

@tuteng Thanks a lot. It worked. Apparantly, I was not including the useTls=true property in functions_worker.yml file.

Just a question what is the difference between using tlsEnabled and useTls in functions_worker.yml?

97arushisharma on 1 Jul 2020

👍1

@97arushisharma

tlsEnabled means enabling TLS in function worker for any incoming requests; useTls means using TLS to connect to broker.

sijie on 2 Jul 2020

Okay! Thanks a lot @sijie .

97arushisharma on 2 Jul 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Failed to decode long type fields in json schema message

lynnmatrix · 3Comments

Websocket example not working - Throwing Unexpected server response: 500

pruthvivrp · 4Comments

Consumer fails to retrieve messages when using multi-host connection with Avro schema and one broker is down in cluster.

kuido85 · 3Comments

Acktimeout cannot be set in Go Client

tfhappy · 3Comments

Add configuration properties to limit the automatic creation of topics.

bilahepan · 4Comments