Pulsar: [pulsar-broker] Policies not found

[devinbost]

We got this exception in our production logs:

Policies not found for myTenant/myNamespace namespace
Reason: Policies not found for myTenant/myNamespace namespace

We were able to reproduce it by running this command:

bin/pulsar-admin --admin-url http://broker12.example.com:8080 topics stats persistent://myTenant/myNamespace/myTopic

However, we were only able to reproduce it by running that command on that particular broker (broker12).

This incident happened at the same time as: https://github.com/apache/pulsar/issues/6492
and https://github.com/apache/pulsar/issues/6054

[devinbost]

I was also told that we were having issues with starting function instances on that same particular broker that was acting up.

[devinbost]

We're also noticing that one of the namespaces intermittently is missing when we run:

bin/pulsar-admin namespaces list myTenant

[lonerzzz]

Was just looking into a similar issue for my own system and saw your post... I identified the problem by connecting with the zookeeper client to the different zookeeper servers. I saw that the appropriate namespace entry was missing under the /namespace path on one of the zookeeper servers. The zookeeper server had lost sync with the others zookeeper servers.

To resolve, I had to clean the snapshot and log files for the out-of=sync zookeeper and restart it. It then resynced with the other servers. The error did not come out after that.

Hope this helps.

[CatherineThompson]

This worked! We found that in one of our zookeeper servers, a couple /admin/policies namespace zkNodes were missing. We followed the above instructions and brought our zookeeper back up with no issues.

[devinbost]

@CatherineThompson Thanks for sharing!
:)