Pulsar: SSL/TLS Configuration for Zookeeper,BookKeeper and Pulsar

[hari819]

hello ,

i want to add ssl/tls configuration to zookeeper using the zookeeper.conf , so first step is modify the the args in the statefulset.yaml file by deleting bin/generate-zookeeper-config.sh conf/zookeeper.conf && cat conf/zookeeper.conf && from the image ,

args: - > bin/wait-for-all-dns.sh && bin/wait-for-my-dns.sh && bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/pulsar zookeeper

if i want to provide zookeeper.conf externally via a configmap , how do i load so that i can add all the ssl configurtion into that file.

the error when starting zookeeper POD with above configuration is ,

> Traceback (most recent call last):
>   File "bin/apply-config-from-env.py", line 44, in <module>
>     for line in open(conf_filename):
> IOError: [Errno 2] No such file or directory: 'conf/pulsar_env.sh

Thanks,

[jiazhai]

@hari819 Seems it missing file conf/pulsar_env.sh. Please check in your env.

[hari819]

thanks @jiazhai @sijie , i could get rid of that error now .
if i have to add some jvm arguments to zookeeper.conf like ,
-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=/root/zookeeper/ssl/testKeyStore.jks -Dzookeeper.ssl.keyStore.password=testpass -Dzookeeper.ssl.trustStore.location=/root/zookeeper/ssl/testTrustStore.jks -Dzookeeper.ssl.trustStore.password=testpass"

where should i add them to add ssl/tls authentication to zookeeper?

[sijie]

@hari819 it might be worth adding a separate page about zookeeper security? All the security related documentation are in this directory, prefixed with "security-" https://github.com/apache/pulsar/blob/master/site2/docs

[hari819]

@sijie , yes that would be a good idea , but if you have the content already with you could you please provide the same here ,?

thanks,

[sijie]

@hari819 I didn't have the content yet. since you already figured this out, it might be worth it for you to contribute a change. thoughts?

[hari819]

@sijie , i can start but need some info on the below ,

i will start by having the zookeeper.conf outside of the the arguments provided for the "pulsar-all" container

args: - > bin/wait-for-all-dns.sh && bin/wait-for-my-dns.sh && bin/apply-config-from-env.py conf/zookeeper.conf && bin/apply-config-from-env.py conf/pulsar_env.sh && bin/generate-zookeeper-config.sh conf/zookeeper.conf && cat conf/zookeeper.conf && bin/pulsar zookeeper

there is a configMap for zookeeper already with kind: ConfigMap metadata: name: zookeeper-config data: PULSAR_MEM: "\" -Xms100m -Xmx256m \"" PULSAR_GC: "\" -XX:+UseG1GC -XX:MaxGCPauseMillis=10\"" , can i start adding it here ?

also this " bin/apply-config-from-env.py conf/zookeeper.conf &&" is the guy who is generating the zookeeper.conf ,

so where to add the ssl properties ?

[hari819]

@sijie , i could get moving on the zookeeper security part , parallel i am also trying to setup tls authentication for bookkeeper , is it supported via pulsar ?.

i followed the docs , https://bookkeeper.apache.org/docs/4.9.0/reference/config/ and https://bookkeeper.apache.org/docs/4.9.0/security/tls/ , but could not get past this error when calling bin/pulsar bookie,

[conf/pulsar_env.sh] Applying config PULSAR_MEM = " -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g -Dpulsar.root.logger=DEBUG,FILE " Error: Could not find or load main class "

will you be able to check this ?

Thanks,

[sijie]

parallel i am also trying to setup tls authentication for bookkeeper , is it supported via pulsar ?.

Yes. it is supported.

but could not get past this error when calling bin/pulsar bookie,

Which version of Pulsar are you using? Try to set BOOKIE_MEM instead of PULSAR_MEM for running bin/pulsar bookie.

[hari819]

@sijie , i am using the latest version of pulsar via the docker image , apachepulsarpulsar-all:latest

[hari819]

parallel i am also trying to setup tls authentication for bookkeeper , is it supported via pulsar ?.

Yes. it is supported.

but could not get past this error when calling bin/pulsar bookie,

Which version of Pulsar are you using? Try to set BOOKIE_MEM instead of PULSAR_MEM for running bin/pulsar bookie.

BOOKIE_MEM solved my problem ,
thanks

[hari819]

@sijie ,
After enabling ssl on zookeepr , i have enabled TLS in bookkeeper also , but i saw some weird error ,

org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://zookeeper/ledgers : zkServers = zookeeper, ledgersRootPath = /ledgers.

why it is , metadata service uri zk+null://zookeeper/ledgers : zkServers = zookeeper, ledgersRootPath = /ledgers

i think the metaformat command is unable to create the folder "/ledgers" ,and finally ends up with bookkeeper going down ,

06:15:35.976 [main] ERROR org.apache.bookkeeper.discover.ZKRegistrationManager - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat
06:15:35.977 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
06:15:35.982 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:392) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
... 10 more

[hari819]

@sijie
, i have now enabled the hierarchical type ,

16:39:52.631 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+hierarchical://zookeeper/ledgers : zkServers = zookeeper, ledgersRootPath = /ledgers.

i am already running the metaformat , using bin/bookkeeper shell metaformat --nonInteractive || true; , before running the bookie,
But still bookkeeper metadata is not available ,
16:39:52.761 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
16:39:52.797 [main] ERROR org.apache.bookkeeper.discover.ZKRegistrationManager - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat
16:39:52.798 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
16:39:52.807 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:392) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:250) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:688) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:105) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.service.BookieService.(BookieService.java:41) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:313) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:382) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
... 10 more

[hari819]

@sijie ,actually the problem was at myside , the shell "bin/bookkeeper shell metaformat --nonInteractive || true;" was missing the zookeeper tls settings ,

i am able to complete SSL/TLS/SASL settings on all the three Zookeeper,Bookkeeper and Pulsar and my pulsar cluster is working fine .

Thanks,

[sijie]

cool! @hari819 are you willing to contribute the documentation for this part?

[hari819]

cool! @hari819 are you willing to contribute the documentation for this part?

@sijie yes , but any guide lines , any document format which i need to follow and prepare

[hari819]

@sijie , could you please provide me a template which i can use to prepare the documentation part ?

[sijie]

@hari819 - you can check this example - https://github.com/apache/pulsar/blob/master/site2/docs/security-authorization.md

[skyrocknroll]

@hari819 We are in the process of deploying the Whole pulsar cluster with TLS. It would be greatly helpful if you document how you have done it. Thanks in advance.

[sijie]

@skyrocknroll

Our helm chart provides the capability of enabling the TLS between components. https://github.com/streamnative/charts You can use them as a reference.

• Configure TLS on bookies:
  https://github.com/streamnative/charts/blob/master/charts/pulsar/templates/bookkeeper/bookkeeper-configmap.yaml#L42
  https://github.com/streamnative/charts/blob/master/charts/pulsar/templates/bookkeeper/_bookkeeper.tpl#L116

• Configure brokers connect to bookies using TLS
  https://github.com/streamnative/charts/blob/master/charts/pulsar/templates/broker/broker-configmap.yaml#L134

• Configure TLS on zookeeper (make sure zookeeper SSL settings are loaded in java system properties when running zookeeper)
  https://github.com/streamnative/charts/blob/master/charts/pulsar/templates/tls/keytool.yaml#L111

• Configure bookkeeper or broker to connect to zookeeper using TLS (make sure zookeeper SSL settings are loaded in java system properties when running zookeeper)
  https://github.com/streamnative/charts/blob/master/charts/pulsar/templates/tls/keytool.yaml#L96

[skyrocknroll]

Thanks @sijie Will use that

[rounak11]

@sijie These github links are not valid anymore, can you please direct us to the correct links. We are trying to enable TLS everywhere and are facing some challenges with ZooKeeper TLS configurations in ZK, broker and bookies.

[sijie]

@rounak11 I have fixed the links. Please check them again.

[hari819]

@sijie @skyrocknroll @rounak11 , did not have time to prepare a document when i completed enabling TLS in February.Hope the following config is helpful bin/pulsar(sh file) and conf files,

ZooKeeper:

elif [ $COMMAND == "zookeeper" ]; then
    PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"zookeeper.log"}
    ZK_OPTS=" -Dzookeeper.4lw.commands.whitelist=* -Dzookeeper.snapshot.trust.empty=true -Djava.security.auth.login.config=conf/zk_jaas.conf -Dzookeeper.requireClientAuthScheme=sasl -Dzookeeper.sasl.client=true -Dzookeeper.sasl.clientconfig=Client -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=keys/KeyStore.jks -Dzookeeper.ssl.keyStore.password=keys/jkspassword -Dzookeeper.ssl.trustStore.location=keys/TrustStore.jks -Dzookeeper.ssl.trustStore.password=keys/jkspassword  -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider"

zookeeper.conf:

secureClientPort=2281

quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer

requireClientAuthScheme=sasl
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

Bookkeeper:

elif [ $COMMAND == "bookie" ]; then
    PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"bookkeeper.log"}
    # Pass BOOKIE_EXTRA_OPTS option defined in pulsar_env.sh
    BOOKIE_EXTRA_OPTS=" -Djavax.net.debug=all -Djavax.net.debug=ssl:handshake:verbose -Dzookeeper.client.secure=true -Djava.security.auth.login.config=conf/bk_jaas.conf"
    OPTS="$OPTS $BOOKIE_EXTRA_OPTS"
    exec $JAVA $OPTS -Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.bookkeeper.proto.BookieServer --conf $PULSAR_BOOKKEEPER_CONF $@

bookkeeper.conf:

`
tlsProvider=OpenSSL

tlsProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory

tlsClientAuthentication=true

tlsEnabledProtocols=TLSv1.2

tlsKeyStoreType=JKS

tlsKeyStore=bookie.keystore.jks

tlsKeyStorePasswordPath=bookie.keystore.passwd

tlsTrustStoreType=JKS

tlsTrustStore=bookie.truststore.jks

tlsTrustStorePasswordPath=bookie.truststore.passwd

clientTrustStore=client.truststore.jks
clientTrustStorePasswordPath=client.truststore.passwd
clientKeyStore=client.keystore.jks
clientKeyStorePasswordPath=client.keystore.passwd`

Pulsar(Broker):

if [ $COMMAND == "broker" ]; then PULSAR_LOG_FILE=${PULSAR_LOG_FILE:-"pulsar-broker.log"} exec $JAVA $OPTS -Djavax.net.debug=all -Djavax.net.debug=ssl:handshake:verbose -Djava.security.auth.login.config=bk_jaas.conf $ASPECTJ_AGENT -Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.pulsar.PulsarBrokerStarter --broker-conf $PULSAR_BROKER_CONF $@

broker.conf

`tlsEnabled=true

tlsCertRefreshCheckDurationSec=300

tlsCertificateFilePath=tls.crt.pem

tlsKeyFilePath=tls.key.pem

tlsTrustCertsFilePath=ca.cert.pem

tlsAllowInsecureConnection=false

tlsProtocols=TLSv1.2,TLSv1.1

tlsRequireTrustedClientCertOnConnect=false

authenticationEnabled=true

authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls

authorizationEnabled=false

authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider

brokerClientTlsEnabled=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters=tlsCertFile:tls.crt.pem,tlsKeyFile:tls.key.pem
brokerClientTrustCertsFilePath=ca.cert.pem

bookkeeperTLSProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory

bookkeeperTLSClientAuthentication=true

bookkeeperTLSKeyFileType=JKS

bookkeeperTLSTrustCertTypes=JKS

bookkeeperTLSKeyStorePasswordPath=bookie.keystore.passwd

bookkeeperTLSTrustStorePasswordPath=bookie.truststore.passwd`

[rounak11]

@rounak11 I have fixed the links. Please check them again.

Is there a way to use PEM keys for ZK?

[hari819]

@rounak11 I have fixed the links. Please check them again.

Is there a way to use PEM keys for ZK?

yes , but from zookeeper 3.5.5 version

[rounak11]

@hari819 Can you please share zk config with PEM configuration.

[hari819]

@rounak11

see if it helps ,

sslQuorum=true serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.quorum.keyStore.location=/certs/certWithPrivateKey.pem ssl.quorum.keyStore.password=******** ssl.quorum.trustStore.location=/path/to/ca/ca-cert.pem ssl.quorum.trustStore.password=****** ssl.hostnameVerification=false ssl.quorum.hostnameVerification=false ssl.keyStore.type=PEM ssl.quorum.keyStore.type=PEM ssl.trustStore.type=PEM ssl.quorum.trustStore.type=PEM

[Ghatage]

@sijie
It seems @hari819's comment has nicely summarized the content to setup Pulsar, Bookkeeper and Zookeeper with TLS.

Do we still need this as a page in the documentation?

Since I recently made a change in documentation, I can take this up and create documentation based on Hari's comment for 'Enabling TLS for Pulsar, Bookkeeper and Zookeeper'.
Let me know.

[sijie]

@Ghatage yeah. I think it would be great if we can get this up in the pulsar documentation website.

[rounak11]

@sijie @hari819 is there a way to enable tls in bookie using pem? If so please let me know the configs. Thanks

[evfurman]

Is there anyone that has successfully configured TLS for zookeeper, bookkeepers, and brokers outside of Kubernetes? Would love to chat in apache-pulsar.slack.com. Username is @evfurman.

Thanks!