Schema-registry: Failed to write Noop record to kafka store

Created on 18 Sep 2015 · 9Comments · Source: confluentinc/schema-registry

I am running two Oracle Linux 6 servers with the full confluent setup (zookeeper, kafka, kafka-rest and schema-registry running on both) and around 80% of the times when I am starting shema registry I get the "Failed to write Noop record to kafka store" error:

Fri Sep 18 19:37:54 UTC 2015: SLF4J: Class path contains multiple SLF4J bindings.
Fri Sep 18 19:37:54 UTC 2015: SLF4J: Found binding in [jar:file:/usr/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
Fri Sep 18 19:37:54 UTC 2015: SLF4J: Found binding in [jar:file:/usr/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
Fri Sep 18 19:37:54 UTC 2015: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Fri Sep 18 19:37:54 UTC 2015: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Fri Sep 18 19:37:54 UTC 2015: [2015-09-18 19:37:54,395] INFO SchemaRegistryConfig values:
Fri Sep 18 19:37:54 UTC 2015: master.eligibility = true
Fri Sep 18 19:37:54 UTC 2015: port = 8081
Fri Sep 18 19:37:54 UTC 2015: kafkastore.timeout.ms = 500
Fri Sep 18 19:37:54 UTC 2015: kafkastore.init.timeout.ms = 60000
Fri Sep 18 19:37:54 UTC 2015: debug = false
Fri Sep 18 19:37:54 UTC 2015: request.logger.name = io.confluent.rest-utils.requests
Fri Sep 18 19:37:54 UTC 2015: metrics.sample.window.ms = 30000
Fri Sep 18 19:37:54 UTC 2015: schema.registry.zk.namespace = schema_registry
Fri Sep 18 19:37:54 UTC 2015: kafkastore.zk.session.timeout.ms = 30000
Fri Sep 18 19:37:54 UTC 2015: kafkastore.topic = _schemas
Fri Sep 18 19:37:54 UTC 2015: avro.compatibility.level = backward
Fri Sep 18 19:37:54 UTC 2015: shutdown.graceful.ms = 1000
Fri Sep 18 19:37:54 UTC 2015: response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
Fri Sep 18 19:37:54 UTC 2015: metrics.jmx.prefix = kafka.schema.registry
Fri Sep 18 19:37:54 UTC 2015: host.name = fr-mktg-ingest01.amers1.cis.trcloud
Fri Sep 18 19:37:54 UTC 2015: metric.reporters = []
Fri Sep 18 19:37:54 UTC 2015: kafkastore.commit.interval.ms = -1
Fri Sep 18 19:37:54 UTC 2015: kafkastore.connection.url = 127.0.0.1:2181,10.206.148.154:2181
Fri Sep 18 19:37:54 UTC 2015: metrics.num.samples = 2
Fri Sep 18 19:37:54 UTC 2015: response.mediatype.default = application/vnd.schemaregistry.v1+json
Fri Sep 18 19:37:54 UTC 2015: kafkastore.topic.replication.factor = 3
Fri Sep 18 19:37:54 UTC 2015: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
Fri Sep 18 19:37:55 UTC 2015: [2015-09-18 19:37:55,059] INFO Initialized the consumer offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:87)
Fri Sep 18 19:37:55 UTC 2015: [2015-09-18 19:37:55,827] WARN The replication factor of the schema topic _schemas is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore:201)
Fri Sep 18 19:37:55 UTC 2015: [2015-09-18 19:37:55,918] INFO [kafka-store-reader-thread-_schemas], Starting  (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:68)
Fri Sep 18 19:38:56 UTC 2015: [2015-09-18 19:38:56,075] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication:57)
Fri Sep 18 19:38:56 UTC 2015: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:164)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:55)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.rest.Application.createServer(Application.java:104)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.rest.Main.main(Main.java:42)
Fri Sep 18 19:38:56 UTC 2015: Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: io.confluent.kafka.schemaregistry.storage.exceptions.StoreException: Failed to write Noop record to kafka store.
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:151)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:162)
Fri Sep 18 19:38:56 UTC 2015: ... 4 more
Fri Sep 18 19:38:56 UTC 2015: Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreException: Failed to write Noop record to kafka store.
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.getLatestOffset(KafkaStore.java:363)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.waitUntilKafkaReaderReachesLastOffset(KafkaStore.java:220)
Fri Sep 18 19:38:56 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:149)
Fri Sep 18 19:38:56 UTC 2015: ... 5 more

At other times it dies right out of the gate with No bootstrap urls given in bootstrap.servers and ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.Main:50)

Fri Sep 18 19:49:51 UTC 2015: SLF4J: Class path contains multiple SLF4J bindings.
Fri Sep 18 19:49:51 UTC 2015: SLF4J: Found binding in [jar:file:/usr/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
Fri Sep 18 19:49:51 UTC 2015: SLF4J: Found binding in [jar:file:/usr/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
Fri Sep 18 19:49:51 UTC 2015: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Fri Sep 18 19:49:51 UTC 2015: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Fri Sep 18 19:49:51 UTC 2015: [2015-09-18 19:49:51,373] INFO SchemaRegistryConfig values:
Fri Sep 18 19:49:51 UTC 2015: master.eligibility = true
Fri Sep 18 19:49:51 UTC 2015: port = 8081
Fri Sep 18 19:49:51 UTC 2015: kafkastore.timeout.ms = 500
Fri Sep 18 19:49:51 UTC 2015: kafkastore.init.timeout.ms = 60000
Fri Sep 18 19:49:51 UTC 2015: debug = true
Fri Sep 18 19:49:51 UTC 2015: request.logger.name = io.confluent.rest-utils.requests
Fri Sep 18 19:49:51 UTC 2015: metrics.sample.window.ms = 30000
Fri Sep 18 19:49:51 UTC 2015: schema.registry.zk.namespace = schema_registry
Fri Sep 18 19:49:51 UTC 2015: kafkastore.zk.session.timeout.ms = 30000
Fri Sep 18 19:49:51 UTC 2015: kafkastore.topic = _schemas
Fri Sep 18 19:49:51 UTC 2015: avro.compatibility.level = backward
Fri Sep 18 19:49:51 UTC 2015: shutdown.graceful.ms = 1000
Fri Sep 18 19:49:51 UTC 2015: response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
Fri Sep 18 19:49:51 UTC 2015: metrics.jmx.prefix = kafka.schema.registry
Fri Sep 18 19:49:51 UTC 2015: host.name = fr-mktg-ingest01.amers1.cis.trcloud
Fri Sep 18 19:49:51 UTC 2015: metric.reporters = []
Fri Sep 18 19:49:51 UTC 2015: kafkastore.commit.interval.ms = -1
Fri Sep 18 19:49:51 UTC 2015: kafkastore.connection.url = 127.0.0.1:2181,10.206.148.154:2181
Fri Sep 18 19:49:51 UTC 2015: metrics.num.samples = 2
Fri Sep 18 19:49:51 UTC 2015: response.mediatype.default = application/vnd.schemaregistry.v1+json
Fri Sep 18 19:49:51 UTC 2015: kafkastore.topic.replication.factor = 3
Fri Sep 18 19:49:51 UTC 2015: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
Fri Sep 18 19:49:52 UTC 2015: [2015-09-18 19:49:52,048] INFO Initialized the consumer offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:87)
Fri Sep 18 19:49:53 UTC 2015: [2015-09-18 19:49:52,995] WARN The replication factor of the schema topic _schemas is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore:201)
Fri Sep 18 19:49:53 UTC 2015: [2015-09-18 19:49:53,099] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.Main:50)
Fri Sep 18 19:49:53 UTC 2015: org.apache.kafka.common.config.ConfigException: No bootstrap urls given in bootstrap.servers
Fri Sep 18 19:49:53 UTC 2015: at org.apache.kafka.common.utils.ClientUtils.parseAndValidateAddresses(ClientUtils.java:46)
Fri Sep 18 19:49:53 UTC 2015: at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:189)
Fri Sep 18 19:49:53 UTC 2015: at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:129)
Fri Sep 18 19:49:53 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:143)
Fri Sep 18 19:49:53 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:162)
Fri Sep 18 19:49:53 UTC 2015: at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:55)
Fri Sep 18 19:49:53 UTC 2015: at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
Fri Sep 18 19:49:53 UTC 2015: at io.confluent.rest.Application.createServer(Application.java:104)
Fri Sep 18 19:49:53 UTC 2015: at io.confluent.kafka.schemaregistry.rest.Main.main(Main.java:42)

Source

zoltan-fedor

Most helpful comment

It seems the issue was caused by problems with the _schema Kafka topic.

I have created a new topic in Kafka called _schema2 and changed the kafkastore.topic parameter in schema-registry.properties to this new schema and since then the Schema Registry does start up reliably.

It would be nice to have better error logging than just saying "Failed to write Noop record to kafka store." when the _schema topic is the one having issues or at least have some pointers in the documentation regarding all the potential causes of this error (as it seems it can be a really wide range of things from not being able to connect to Kafka or zookeepe broker connection issues.

zoltan-fedor on 9 Oct 2015

🎉3 👍2

All 9 comments

I have the "noop" issue 100% of the time today after not seeing it at all yesterday.

jdfree on 18 Sep 2015

@jdfree somehow I am back to 100% too, this error seems to come up every single time now.

The same time when I get these "noop" issues from the Schema Registry, zookeeper registers end of session warnings, but I'm pretty sure that is just to register the fact that Schema Registry's request has timed out and not the "cause" of this issue.
In any case I thought I share it:

Fri Sep 18 20:42:04 UTC 2015: [2015-09-18 20:42:04,799] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
Fri Sep 18 20:42:04 UTC 2015: EndOfStreamException: Unable to read additional data from client sessionid 0x14fe1f37fea001b, likely client has closed socket
Fri Sep 18 20:42:04 UTC 2015: at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Fri Sep 18 20:42:04 UTC 2015: at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
Fri Sep 18 20:42:04 UTC 2015: at java.lang.Thread.run(Thread.java:701)
Fri Sep 18 20:42:04 UTC 2015: [2015-09-18 20:42:04,800] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
Fri Sep 18 20:42:04 UTC 2015: EndOfStreamException: Unable to read additional data from client sessionid 0x14fe1f37fea001d, likely client has closed socket
Fri Sep 18 20:42:04 UTC 2015: at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Fri Sep 18 20:42:04 UTC 2015: at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
Fri Sep 18 20:42:04 UTC 2015: at java.lang.Thread.run(Thread.java:701)

zoltan-fedor on 18 Sep 2015

I have verified that the error is kafka-side.

jdfree on 21 Sep 2015

Hi @zoltan-fedor and @jdfree

To date, we've seen two causes of this type of issue:

First, connectivity issues - if schema registry can't open tcp connections to zookeeper and kafka brokers, it will not be able to start. Can you verify the you can open tcp connections to kafka brokers and zookeeper nodes from the schema registry machine(s)?

Try:

    telnet <host> <port>

Also, given that broker.list is empty, there may be a problem on the Kafka server side, e.g. the broker process(es) failed to start, possibly due to connectivity issues to your zookeeper cluster.

Second, if you are running Kafka brokers v0.8.1 or earlier, there is a known compatibility issue which causes a StoreTimeoutException:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/confluent-platform/Q-xTXzpdzc4/ra9jDvfYDSIJ

granders on 21 Sep 2015

I have created services out of Kafka, Zookeeper and the Schema Registry and it seems the problem was that they might start too quick, so Zookeeper is not yet fully ready to receive and cause the Kafka brokers to fail and so the Schema Registry doesn't start either.
Adding delays solves it.

zoltan-fedor on 2 Oct 2015

Well, the next time I was restarting the Schema Registry the same issue came back again.

Yes, I can connect to the zookeeper issues (all 3 of them).
There is only one kafka instance running, but that doesn't report any issues and Kafka Rest has connected to it fine.

Kafka only logs the fact that it lost the socket connection from the Schema registry, nothing else.

Below are the different errors:

Schema Registry (10.206.148.154):

Fri Oct  9 19:21:29 UTC 2015: SLF4J: Class path contains multiple SLF4J bindings.
Fri Oct  9 19:21:29 UTC 2015: SLF4J: Found binding in [jar:file:/usr/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
Fri Oct  9 19:21:29 UTC 2015: SLF4J: Found binding in [jar:file:/usr/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
Fri Oct  9 19:21:29 UTC 2015: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Fri Oct  9 19:21:29 UTC 2015: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Fri Oct  9 19:21:29 UTC 2015: [2015-10-09 19:21:29,577] INFO SchemaRegistryConfig values:
Fri Oct  9 19:21:29 UTC 2015: master.eligibility = true
Fri Oct  9 19:21:29 UTC 2015: port = 8081
Fri Oct  9 19:21:29 UTC 2015: kafkastore.timeout.ms = 500
Fri Oct  9 19:21:29 UTC 2015: kafkastore.init.timeout.ms = 60000
Fri Oct  9 19:21:29 UTC 2015: debug = false
Fri Oct  9 19:21:29 UTC 2015: request.logger.name = io.confluent.rest-utils.requests
Fri Oct  9 19:21:29 UTC 2015: metrics.sample.window.ms = 30000
Fri Oct  9 19:21:29 UTC 2015: schema.registry.zk.namespace = schema_registry
Fri Oct  9 19:21:29 UTC 2015: kafkastore.zk.session.timeout.ms = 30000
Fri Oct  9 19:21:29 UTC 2015: kafkastore.topic = _schemas
Fri Oct  9 19:21:29 UTC 2015: avro.compatibility.level = backward
Fri Oct  9 19:21:29 UTC 2015: shutdown.graceful.ms = 1000
Fri Oct  9 19:21:29 UTC 2015: response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
Fri Oct  9 19:21:29 UTC 2015: metrics.jmx.prefix = kafka.schema.registry
Fri Oct  9 19:21:29 UTC 2015: host.name = xxx.xxx.xx
Fri Oct  9 19:21:29 UTC 2015: metric.reporters = []
Fri Oct  9 19:21:29 UTC 2015: kafkastore.commit.interval.ms = -1
Fri Oct  9 19:21:29 UTC 2015: kafkastore.connection.url = 10.206.148.154:2181,10.206.77.136:2181,10.206.77.14:2181
Fri Oct  9 19:21:29 UTC 2015: metrics.num.samples = 2
Fri Oct  9 19:21:29 UTC 2015: response.mediatype.default = application/vnd.schemaregistry.v1+json
Fri Oct  9 19:21:29 UTC 2015: kafkastore.topic.replication.factor = 3
Fri Oct  9 19:21:29 UTC 2015: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
Fri Oct  9 19:21:30 UTC 2015: [2015-10-09 19:21:30,114] INFO Initialized the consumer offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:87)
Fri Oct  9 19:21:30 UTC 2015: [2015-10-09 19:21:30,843] WARN The schema topic _schemas should have only 1 partition. (io.confluent.kafka.schemaregistry.storage.KafkaStore:197)
Fri Oct  9 19:21:30 UTC 2015: [2015-10-09 19:21:30,844] WARN The replication factor of the schema topic _schemas is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore:201)
Fri Oct  9 19:21:30 UTC 2015: [2015-10-09 19:21:30,925] INFO [kafka-store-reader-thread-_schemas], Starting  (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:68)
Fri Oct  9 19:22:31 UTC 2015: [2015-10-09 19:22:31,109] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication:57)
Fri Oct  9 19:22:31 UTC 2015: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:164)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:55)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.rest.Application.createServer(Application.java:104)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.rest.Main.main(Main.java:42)
Fri Oct  9 19:22:31 UTC 2015: Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: io.confluent.kafka.schemaregistry.storage.exceptions.StoreException: Failed to write Noop record to kafka store.
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:151)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:162)
Fri Oct  9 19:22:31 UTC 2015: ... 4 more
Fri Oct  9 19:22:31 UTC 2015: Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreException: Failed to write Noop record to kafka store.
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.getLatestOffset(KafkaStore.java:363)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.waitUntilKafkaReaderReachesLastOffset(KafkaStore.java:220)
Fri Oct  9 19:22:31 UTC 2015: at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:149)
Fri Oct  9 19:22:31 UTC 2015: ... 5 more

Kafka (10.206.148.154):

Fri Oct  9 19:22:31 UTC 2015: [2015-10-09 19:22:31,434] ERROR Closing socket for /10.206.148.154 because of error (kafka.network.Processor)
Fri Oct  9 19:22:31 UTC 2015: java.io.IOException: Connection reset by peer
Fri Oct  9 19:22:31 UTC 2015: at sun.nio.ch.FileDispatcher.read0(Native Method)
Fri Oct  9 19:22:31 UTC 2015: at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
Fri Oct  9 19:22:31 UTC 2015: at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
Fri Oct  9 19:22:31 UTC 2015: at sun.nio.ch.IOUtil.read(IOUtil.java:224)
Fri Oct  9 19:22:31 UTC 2015: at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
Fri Oct  9 19:22:31 UTC 2015: at kafka.utils.Utils$.read(Utils.scala:380)
Fri Oct  9 19:22:31 UTC 2015: at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
Fri Oct  9 19:22:31 UTC 2015: at kafka.network.Processor.read(SocketServer.scala:444)
Fri Oct  9 19:22:31 UTC 2015: at kafka.network.Processor.run(SocketServer.scala:340)
Fri Oct  9 19:22:31 UTC 2015: at java.lang.Thread.run(Thread.java:701)

Zookeeper 1 (10.206.148.154):

Fri Oct  9 19:21:29 UTC 2015: [2015-10-09 19:21:29,988] WARN Connection request from old client /10.206.148.154:36290; will be dropped if server is in r-o mode (org.apache.zookeeper.server.ZooKeeperServer)
Fri Oct  9 19:22:31 UTC 2015: [2015-10-09 19:22:31,433] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
Fri Oct  9 19:22:31 UTC 2015: EndOfStreamException: Unable to read additional data from client sessionid 0x2504df3e0030004, likely client has closed socket
Fri Oct  9 19:22:31 UTC 2015: at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Fri Oct  9 19:22:31 UTC 2015: at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
Fri Oct  9 19:22:31 UTC 2015: at java.lang.Thread.run(Thread.java:701)

Zookeeper 2:

Fri Oct  9 15:21:30 EDT 2015: [2015-10-09 15:21:30,364] WARN Connection request from old client /10.206.148.154:57651; will be dropped if server is in r-o mode (org.apache.zookeeper.server.ZooKeeperServer)
Fri Oct  9 15:22:31 EDT 2015: [2015-10-09 15:22:31,439] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
Fri Oct  9 15:22:31 EDT 2015: EndOfStreamException: Unable to read additional data from client sessionid 0xb504d9adcb10008, likely client has closed socket
Fri Oct  9 15:22:31 EDT 2015: at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
Fri Oct  9 15:22:31 EDT 2015: at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

Zookeeper 3:
No logs from this period

zoltan-fedor on 9 Oct 2015

It seems the issue was caused by problems with the _schema Kafka topic.

zoltan-fedor on 9 Oct 2015

🎉3 👍2

I am getting the same error messages after upgrading to Schema Registry 2.0.1 from 1.0. I followed the upgrade documentation on the Confluent site; I upgraded _all_ of the brokers before the schema. More information about the schema registry during the upgrade process would be helpful.

There seems to be something wrong with my existing _schema topic now as well. Changing to a new topic got the server to start, but I'd prefer not to have to do the same in production.

enak on 23 Mar 2016

I am having the same issue. I tried using a new topic _schemas2 instead of _schemas. But, it didn't help.
I only have one-node kafka.

We found that it may be caused by one of the broker endpoints discovered by the schem-registry being inaccessible. After removing that kafka server from adveristing, the issue is gone.