Schema-registry: Get NullPointerException when schema topic is not 1 partition

[sweat123]

[2017-12-15 04:03:21,316] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry)
[2017-12-15 04:03:21,320] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
java.lang.NullPointerException
at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:66)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:42)
at io.confluent.rest.Application.createServer(Application.java:157)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

Method verifySchemaTopic() throw StoreInitializationException when I used a schema topic with 3 partition;It may seem strange I will get a NullPointerException instead of wrong message.

[sweat123]

schema registry version 4.0.0;

1. Method createOrVerifySchemaTopic() in KafkaStore.class#init() will throw StoreInitializationException and it is catched in KafkaSchemaRegistry.class#init(), then throw a SchemaRegistryException and is catched in SchemaRegistryRestApplication.class#setupResources(), finally onShutdown() will be called.

    try {
      schemaRegistry = new KafkaSchemaRegistry(
          schemaRegistryConfig,
          new SchemaRegistrySerializer()
      );
      schemaRegistry.init();
    } catch (SchemaRegistryException e) {
      onShutdown();
      log.error("Error starting the schema registry", e);
      System.exit(1);
    }

1. Then we will call schemaRegistry.close() which will call kafkaStore.close() in KafkaSchemaRegistry.class;

  @Override
  public void onShutdown() {

    if (schemaRegistry != null) {
      schemaRegistry.close();
    }
    ...

  @Override
  public void close() {
    log.info("Shutting down schema registry");
    kafkaStore.close();
    ...



md5-63c9f106615b3108284bb143b3699d92



```java
  @Override
  public void init() throws StoreInitializationException {
    ...
    createOrVerifySchemaTopic();
    ...
    producer = new KafkaProducer<byte[], byte[]>(props);
    ...
    this.kafkaTopicReader =
        new KafkaStoreReaderThread<>(this.bootstrapBrokers, topic, groupId,
                                     this.storeUpdateHandler, serializer, this.localStore,
                                     this.noopKey, this.config);
    this.kafkaTopicReader.start();
    ...

[jonbuffington]

@sweat123 I ran into the same issue where the NPE was masking the underlying issue. My fix was similar to yours except I also added a test for localStore:

...
  if (localStore != null) {
    localStore.close();
    log.debug("Kafka store shut down complete");
  }
}

[honnel]

We also run into this NPE issue. As workaround we created the _schemas-Topic manually with only one partition before startup of schema-registry.

any plans for next release date?

[dancarroll67]

I am getting this same issue. I created "_schemas" manually, started confluent again, but schema_registry still goes down with the same npe. Will there be an update/ release to fix this?

[dancarroll67]

Can anyone help with this issue? I am on Mac 10.11.6 and java 1.8.0_131_-b11. I download Confluent open source (confluent-4.0.0), untar it, and run ./confluent start. I get the error above. Two of my teammates go through the same procedure and do not get the error above.

[sweat123]

@dancarroll67 May be there is something wrong with your topic configuration. You need set your topic "_schemas" with only 1 partition. Schema Registry will verify your schema topic in KafkaStore.verifySchemaTopic(AdminClient admin).

[honnel]

It's currently scheduled to be in the 4.1 release, which is planned for late February (via @confluentinc Twitter account).

[honnel]

Workaround:

• Make sure all instances of schema-registry in cluster are shutdown.
• Delete Schema Topic:

kafka-topics --zookeeper localhost:2181 --delete --topic _schemas

• Make shure deletion has taken place (can take long time, see https://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed)
• Create topic with following settings:

kafka-topics --create --zookeeper localhost:2181 --topic _schemas --replication-factor 3 --partitions 1 --config cleanup.policy=compact

[tarekabouzeid]

hi honnel, i tried creating the topic as you suggested but still having same issue

[sweat123]

@tarekabouzeid Maybe there is something wrong with your configuration. Could you show me your topic configuration?

[tarekabouzeid]

@sweat123 i used the below command line to create the topic:
kafka-topics --create --zookeeper localhost:2181 --topic _schemas --replication-factor 3 --partitions 1 --config cleanup.policy=compact

and used the below command to start the schema registry:
./schema-registry-start /etc/schema-registry.properties

and in file schema-registry.properties, below is my configuration:
listeners=http://0.0.0.0:8081
kafkastore.connection.url=localhost:2181
kafkastore.topic=_schemas
debug=false

running on java 1.8.0_161-b12
OS: RHEL 7.4

[sweat123]

@tarekabouzeid seems no problem. Could you provide exception information?

[tarekabouzeid]

@sweat123 please check the below

[root@drbdmnsitap1v bin]# ./schema-registry-start /usr/software/confluent-4.0.0/etc/schema-registry/schema-registry.properties
[2018-02-27 13:25:57,091] INFO SchemaRegistryConfig values:
metric.reporters = []
kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit
response.mediatype.default = application/vnd.schemaregistry.v1+json
kafkastore.ssl.trustmanager.algorithm = PKIX
authentication.realm =
ssl.keystore.type = JKS
kafkastore.topic = _schemas5
metrics.jmx.prefix = kafka.schema.registry
kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
kafkastore.topic.replication.factor = 3
ssl.truststore.password = [hidden]
kafkastore.timeout.ms = 500
host.name = drbdmnsitap1v.lbdc
kafkastore.bootstrap.servers = []
schema.registry.zk.namespace = schema_registry
kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8
kafkastore.sasl.kerberos.service.name =
schema.registry.resource.extension.class =
ssl.endpoint.identification.algorithm =
compression.enable = false
kafkastore.ssl.truststore.type = JKS
avro.compatibility.level = backward
kafkastore.ssl.protocol = TLS
kafkastore.ssl.provider =
kafkastore.ssl.truststore.location =
response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
kafkastore.ssl.keystore.type = JKS
ssl.truststore.type = JKS
kafkastore.ssl.truststore.password = [hidden]
access.control.allow.origin =
ssl.truststore.location =
ssl.keystore.password = [hidden]
port = 8081
kafkastore.ssl.keystore.location =
master.eligibility = true
ssl.client.auth = false
kafkastore.ssl.keystore.password = [hidden]
kafkastore.security.protocol = PLAINTEXT
ssl.trustmanager.algorithm =
authentication.method = NONE
request.logger.name = io.confluent.rest-utils.requests
ssl.key.password = [hidden]
kafkastore.zk.session.timeout.ms = 30000
kafkastore.sasl.mechanism = GSSAPI
kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05
kafkastore.ssl.key.password = [hidden]
zookeeper.set.acl = false
schema.registry.inter.instance.protocol = http
authentication.roles = [*]
metrics.num.samples = 2
ssl.protocol = TLS
schema.registry.group.id = schema-registry
kafkastore.ssl.keymanager.algorithm = SunX509
kafkastore.connection.url = localhost:2181
debug = true
listeners = [http://0.0.0.0:8081]
kafkastore.group.id =
ssl.provider =
ssl.enabled.protocols = []
shutdown.graceful.ms = 1000
ssl.keystore.location =
ssl.cipher.suites = []
kafkastore.ssl.endpoint.identification.algorithm =
kafkastore.ssl.cipher.suites =
access.control.allow.methods =
kafkastore.sasl.kerberos.min.time.before.relogin = 60000
ssl.keymanager.algorithm =
metrics.sample.window.ms = 30000
kafkastore.init.timeout.ms = 60000
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:175)
[2018-02-27 13:25:57,294] DEBUG Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:176)
[2018-02-27 13:25:57,296] INFO Logging initialized @490ms (org.eclipse.jetty.util.log:186)
[2018-02-27 13:25:57,659] DEBUG Kafka store zookeeper client shut down (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:549)
[2018-02-27 13:25:57,660] INFO Initializing KafkaStore with broker endpoints: PLAINTEXT://drbddnsitap3v.lbdc:9092,PLAINTEXT://drbddnsitap2v.lbdc:9092,PLAINTEXT://drbddnsitap1v.lbdc:9092 (io.confluent.kafka.schemaregistry.storage.KafkaStore:103)
[2018-02-27 13:25:57,890] INFO Validating schemas topic _schemas5 (io.confluent.kafka.schemaregistry.storage.KafkaStore:228)
[2018-02-27 13:25:57,903] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:719)
[2018-02-27 13:25:57,904] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)
java.lang.NullPointerException
at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:66)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:42)
at io.confluent.rest.Application.createServer(Application.java:157)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

[sweat123]

@tarekabouzeid Your kafkastore.topic is "_schemas5";

1. "_schemas5" has only 1 partition?
2. Your kafka version is 1.0.0?
3. your kafka run in localhost?

[tarekabouzeid]

1- yes, 1 partition with 3 replicas
2- Kafka version is 2.1.0
3- zookeeper is on localhost, but the actual kafka are on different machines.

[sweat123]

@tarekabouzeid

[2018-02-27 13:25:57,890] INFO Validating schemas topic _schemas5 (io.confluent.kafka.schemaregistry.storage.KafkaStore:228)
[2018-02-27 13:25:57,903] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:719)

It seems the configuration topic is not correct;
SchemaRegistry will verify schema topic with function KafkaStore.verifySchemaTopic; Maybe you can clone the repo and find what is wrong with your configuration;
I can't found out why you will get NPE with listed information; so sorry!

[tarekabouzeid]

@sweat123 when i shutdown my cloudera zookeeper and kafka brokers, and use this command:
confluent start schema-registry
all services will work fine, so can you please guide me how to make schema-registry run using my already running zookeeper and kafka brokers?

[slepkin]

I'm surprised that the "official" workaround for this involves deleting the _schemas topic; doesn't that make it impossible to read all AVRO messages that were written using the Registry?

[terry19850829]

I have the same issue , any idea to fix this ??
I have delete the _schemas topic by zkcli , clean the topic data and recreate.
Still have this issue...

[charsyam]

@terry19850829 Did you check kafka version and confluent version? You have to match version of them.

[terry19850829]

@charsyam I have resolve this issue by cleaning all my kafka data. I've just tested confluent platform for using. Some data should be deteled have not been recorded in zk ?

[anderseriksson]

It can also be a permission issue if you are using Kafka with ACL. When Schema Registry is validating the _schemas topic it need DESCRIBE permission (not only consume/produce) so we resolved this problem by the following

kafka-acls --add --allow-principal User:"CN=schemaregistry" --group="*" --operation All --topic="_schemas" --authorizer-properties zookeeper.connect=zookeeper01:2181

[mparthas]

Confluent 5.0.1 has the same problem when I just use my local mac for installation. I have created _schemas and it still does not work.. Any solution for this ?

[rayokota]

The NPE described in the original problem was fixed in 4.1.x