Pulsar: when authentication enabled pulsar-admin topics list will fail for nonpersistent topics

[jiazhai]

Describe the bug
when user enable authentication, pulsar-admin topics list will fail.

in the code path, persistent topic list could pass the auth, but nonPersistent topic will fail with error code 401

other command like tenants/namespaces list works fine.

This could reproduce when authentication is enabled

[jiazhai]

looking and debugging into the code.

this issue happens under this condition

1. broker enable authentication.
2. broker not set ”brokerClientAuthenticationPlugin" and "brokerClientAuthenticationParameters“

this issue happens like this

1. user use pulsar-admin topics list to get all the topics, and internally this command will send request for both persistent and non-persistent rest endpoint.
2. for persistent rest endpoint, it runs well, since all the topics information could be get from /managed-ledgers in zk.
3. while for non-persistent, it will first get all the bundles of this namespace, and these bundles may spread across different brokers, so for each bundle, it will call
   pulsar().getAdminClient().topics().getListInBundleAsync(namespaceName.toString(), bundle)
   to get the topic list in each bundle.
4. in method getAdminClient(), it will create an internal pulsar admin client, and its authentication method is determined by parameter ”brokerClientAuthenticationPlugin" and "brokerClientAuthenticationParameters“.

public synchronized PulsarAdmin getAdminClient() throws PulsarServerException {
        if (this.adminClient == null) {
                ServiceConfiguration conf = this.getConfiguration();
                String adminApiUrl = conf.isBrokerClientTlsEnabled() ? webAddressTls(config) : webAddress(config);
                PulsarAdminBuilder builder = PulsarAdmin.builder().serviceHttpUrl(adminApiUrl) //
                        .authentication( //
                                conf.getBrokerClientAuthenticationPlugin(), //   < === 
                                conf.getBrokerClientAuthenticationParameters());  < ===
...
                builder.readTimeout(conf.getZooKeeperOperationTimeoutSeconds(), TimeUnit.SECONDS);
                this.adminClient = builder.build();
            } 

        return this.adminClient;
    }

since auth parameter not provided, so it will return 401, as in the picture.

[jiazhai]

To fix this issue, user need to provide settings of ”brokerClientAuthenticationPlugin" and "brokerClientAuthenticationParameters“ in broker.conf

[sijie]

@jiazhai do we need to improve the documentation for this?

[Liu-Junlin]

To fix this issue, user need to provide settings of ”brokerClientAuthenticationPlugin" and "brokerClientAuthenticationParameters“ in broker.conf

@jiazhai
Could you tell me how to config this,please.
I have done this by:

1.config a token Authentication:

brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///data/pulsar/apache-pulsar/conf/secret.key

2.set a token when build admin client:

PulsarAdmin.builder()
.authentication("org.apache.pulsar.client.impl.auth.AuthenticationToken",token)
.serviceHttpUrl(pulsarProperties.getServiceUrl())
.build()

But the result always return bad reques status,the pulsar broker server log is :
java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException: HTTP 400 Bad Request at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_202] at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:184) ~[org.apache.pulsar-pulsar-broker-2.4.0.jar:2.4.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202] at
pulsar-broker-cloud-1.log

[Liu-Junlin]

To fix this issue, user need to provide settings of ”brokerClientAuthenticationPlugin" and "brokerClientAuthenticationParameters“ in broker.conf

@jiazhai
Could you tell me how to config this,please.
I have done this by:

1.config a token Authentication:

brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken brokerClientAuthenticationParameters=file:///data/pulsar/apache-pulsar/conf/secret.key

2.set a token when build admin client:

PulsarAdmin.builder() .authentication("org.apache.pulsar.client.impl.auth.AuthenticationToken",token) .serviceHttpUrl(pulsarProperties.getServiceUrl()) .build()

But the result always return bad reques status,the pulsar broker server log is :
java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException: HTTP 400 Bad Request at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_202] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_202] at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:184) ~[org.apache.pulsar-pulsar-broker-2.4.0.jar:2.4.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_202] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_202] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202] at
pulsar-broker-cloud-1.log

I have found my problem.
The config brokerClientAuthenticationParameters must be a client_secret,like a token.
brokerClientAuthenticationParameters=token:eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJwdWxzYXItYWRtaW4ifQ.k2Fa1ZxDJTUHBGPsxLjT7x4hJ9Cvt_19DBbwsX-URdk
Then everything is OK.

[xixiss]

i write these in broker.conf:
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters=tlsCertFile:/root/my-ca/broker.cert.pem,tlsKeyFile:/root/my-ca/broker.key-pk8.pem
brokerClientTrustCertsFilePath=/root/my-ca/certs/ca.cert.pem

but it still fail with error code 401 when i use pulsar-admin to list topic.And the TLS is work well in other situation.

[jiazhai]

Sorry for not notice this. @Liu-Junlin @xixiss.
@sijie We should add doc. But after re-think of this, we may should provide a config for user to disable the auth between brokers.
currently this config is re-used at a lot of places,

brokerClientAuthenticationPlugin=
brokerClientAuthenticationParameters=

Broker internal client, Broker internal admin, Broker internal function worker will all use this. If we want to use this config in more than 1 situation, e.g. enable Broker internal client and Broker internal function worker at the same time ,but with different config, it is impossible.

[tuteng]

I will try to add more documents to fix these problems.