Schema-registry: Got "ERROR The retention policy of the schema topic _schemas is incorrect"?

[namjitharavind]

Hi,

I have 4 machines among this 3 brokers have installed both zookeeper and kafka and in remain one machine installed kafka-rest and schema-registry. Below is the schema registry properties.

listeners=http://0.0.0.0:8081
kafkastore.connection.url=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka
kafkastore.topic=_schemas
debug=false

But when i try to start schema registry

$CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties
i am getting the following error.

ERROR The retention policy of the schema topic _schemas is incorrect. You must configure the topic to 'compact' cleanup policy to avoid Kafka deleting your schemas after a week. Refer to Kafka documentation for more details on cleanup policies (io.confluent.kafka.schemaregistry.storage.KafkaStore:388)
[2017-12-28 05:04:48,102] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:52)
java.lang.IllegalStateException: The retention policy of the schema topic _schemas is incorrect. Expected cleanup.policy to be 'compact' but it is null
at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:393)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:283)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:152)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:212)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:56)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:38)
at io.confluent.rest.Application.createServer(Application.java:157)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)

[balboah]

Similar error (docker image v4.0.0)

[2018-01-11 14:37:49,682] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2018-01-11 14:37:49,773] ERROR The retention policy of the schema topic _schemas is incorrect. You must configure the topic to 'compact' cleanup policy to avoid Kafka deleting your schemas after a week. Refer to Kafka documentation for more details on cleanup policies (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2018-01-11 14:37:49,780] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry)
[2018-01-11 14:37:49,783] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
java.lang.NullPointerException
    at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366)
    at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720)
    at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111)

I'm thinking it only happens sometimes and perhaps after restarting kafka cluster or when schema-registry is restarting often.

Manually fixed with kafka-topics --alter --config cleanup.policy=compact --topic _schemas --zookeeper dev-zookeeper/kafka

[lwu08]

anyone knows the root cause? encounter this issue too ...

[gmyrianthous]

@lwu08 One reason that might cause this problem is when you accidentally delete _schemas topic.

[VeselaHouba]

kafka-topics is deprecated
You should use

kafka-configs.sh --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact

Still only workaround

[gopikrishnasilla]

below command worked for me .. thanks bud ...
kafka-configs.sh --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact

[keithhawkridge]

We are using helm to deploy kafka,zookeeper and schema registry at the same time

The majority of time, the initial startup attempt of schema-registry throws the following error

[2019-12-19 02:22:56,301] WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2019-12-19 02:22:56,304] INFO Kafka version: 5.3.1-ce (org.apache.kafka.common.utils.AppInfoParser) [2019-12-19 02:22:56,304] INFO Kafka commitId: 30df4bec2f6882ab (org.apache.kafka.common.utils.AppInfoParser) [2019-12-19 02:22:56,304] INFO Kafka startTimeMs: 1576722176301 (org.apache.kafka.common.utils.AppInfoParser) [2019-12-19 02:23:00,352] INFO Creating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2019-12-19 02:23:01,849] WARN Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2019-12-19 02:23:03,554] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2019-12-19 02:23:04,561] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication) io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:224) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:66) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:81) at io.confluent.rest.Application.createServer(Application.java:205) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:42) Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: Failed trying to create or validate schema topic configuration at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:179) at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:116) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:222) ... 4 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272) at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:237) at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:222) at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:171) ... 6 more Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

Subsequent restarts of schema reg, throws the error in the first comment

And there again, sometimes it works fine.

[selfeky]

Is there a solution for this?
It happens when schema registry restarts

[sherry-ummen]

We are using helm to deploy kafka,zookeeper and schema registry at the same time

The majority of time, the initial startup attempt of schema-registry throws the following error

[2019-12-19 02:22:56,304] INFO Kafka version: 5.3.1-ce (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-19 02:22:56,304] INFO Kafka commitId: 30df4bec2f6882ab (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-19 02:22:56,304] INFO Kafka startTimeMs: 1576722176301 (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-19 02:23:00,352] INFO Creating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2019-12-19 02:23:01,849] WARN Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2019-12-19 02:23:03,554] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2019-12-19 02:23:04,561] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)
io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry
        at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:224)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:66)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:81)
        at io.confluent.rest.Application.createServer(Application.java:205)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:42)
Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: Failed trying to create or validate schema topic configuration
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:179)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:116)
        at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:222)
        ... 4 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:237)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:222)
        at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:171)
        ... 6 more
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.```

Subsequent restarts of schema reg, throws the error in the first comment

And there again, sometimes it works fine.

Ditto, this is how we deploy too, and this is pain in the * , everytime registry crashes or reboots we end up in this situation, and how come this issues still a thing even after 3 years. Really sad state of this project.

[alok87]

Is there a solution for this?
It happens when schema registry restarts

Is there a permanent solution for this? It is happening on every schema registry restart