Sarama: Reading Azure event hubs fails with "kafka: response did not contain all the expected topic/partition blocks"

[faec]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

| Sarama | Kafka | Go |
|--------|-------|----|
| 1.24.1 | 1.0 | 1.12.4 |

Configuration

What configuration values are you using for Sarama and Kafka?

k := sarama.NewConfig()

k.Version = sarama.V1_0_0_0

k.Consumer.Return.Errors = true
k.Consumer.Offsets.Retry.Max = 5
k.Consumer.Offsets.Initial = sarama.OffsetOldest

k.Net.TLS.Enable = true
k.Net.SASL.Enable = true
k.Net.SASL.User = "$ConnectionString"
k.Net.SASL.Password = connectionStr

Logs

When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.

logs: CLICK ME

[Sarama] 2019/11/15 13:52:56 Initializing new client
[Sarama] 2019/11/15 13:52:56 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[Sarama] 2019/11/15 13:52:56 ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[Sarama] 2019/11/15 13:52:56 client/metadata fetching metadata for all topics from broker filebeat-test.servicebus.windows.net:9093
[Sarama] 2019/11/15 13:52:56 Successful SASL handshake. Available mechanisms: [PLAIN]
[Sarama] 2019/11/15 13:52:56 SASL authentication successful with broker filebeat-test.servicebus.windows.net:9093:4 - [0 0 0 0]
[Sarama] 2019/11/15 13:52:56 Connected to broker at filebeat-test.servicebus.windows.net:9093 (unregistered)
[Sarama] 2019/11/15 13:52:57 client/brokers registered new broker #0 at filebeat-test.servicebus.windows.net:9093
[Sarama] 2019/11/15 13:52:57 Successfully initialized new client
[Sarama] 2019/11/15 13:52:57 client/metadata fetching metadata for [kafka-west] from broker filebeat-test.servicebus.windows.net:9093
[Sarama] 2019/11/15 13:52:57 client/coordinator requesting coordinator for consumergroup $Default from filebeat-test.servicebus.windows.net:9093

Problem Description

(This is arguably an issue in Azure rather than sarama, but I'm sharing it since it potentially affects a lot of sarama users and might require a workaround. It doesn't depend on the kafka or sarama version.)

Starting recently, Azure's kafka implementation ("event hubs") started responding to OffsetFetchRequest for new consumer groups with an OffsetFetchResponse containing an empty offset table. This causes sarama clients to fail when fetching the initial offset from the coordinator, with the error message kafka: response did not contain all the expected topic/partition blocks.

As described in this spec page, we expect a response of the form [TopicName [Partition Offset Metadata ErrorCode]]. For topic / partition pairs that have not been read, we expect the inner partition table to contain empty metadata and an offset of -1. This is what Azure event hubs used to do, but early this week we started seeing failures that we traced to responses with an empty partition table.

The failure in sarama happens in offset_manager.go:fetchInitialOffset. The offset is requested with resp, err := broker.FetchOffset(req), which succeeds. The broker returns a valid OffsetFetchResponse, but the block table inside is empty, which causes the following call block := resp.GetBlock(topic, partition) to fail.

As far as we can tell, this change only affects consumer groups that have never committed an explicit offset. We have a debug workaround that fixes the problem by inserting a placeholder offset when the response table is empty -- after that, the event hub can be read, and as soon as the consumer commits an offset the problem goes away, and future connections will get a nonempty offset response.

We don't know yet if this is an intentional change on Azure's part (we have shared the issue with them and are waiting for more information), or if there is some subtlety of the Kafka spec that allows for an empty table in new consumer groups. If Azure reverts to the old behavior, this issue may be moot, but in the meantime I wanted to share what we've found in case others are encountering the same problem.

[chrisheunis]

Thanks for this info, It's very helpful. I've run into a similar issue with the Telegraf kafka_consumer plugin, which uses the Sarama library.
I've also logged an support request with Microsoft about this issue, will see what feedback I get.

[faec]

I've posted more details, including a capture of the raw protocol messages exhibiting the problem, to this issue

[arerlend]

This should be fixed on the EH side now. Will update the thread when the fix reaches deployment.

[chrisheunis]

That's great news. Thanks for the update Arthur.

[bai]

Nice. Assuming https://github.com/Shopify/sarama/pull/1542 is no longer relevant?

[arerlend]

Yeah - this was a regression on our end, it is not correct behavior and shouldn't be added. A retry mechanism was removed and partitions with no stored offsets or only expired offsets had nothing returned.

We didn't catch it in our stress tests since the Java client (what we consider reference impl) unintentionally handles the empty case gracefully.

[jonnylangefeld]

@arerlend, how will an eventhubs instance in production with no committed offset behave? Will we have to recreate it after the new instance was deployed? Or will i be updated automatically behind the scenes?

[arerlend]

On the first question - @faec did a great job detailing the bugged behavior. (Thanks!!)

To get the fix, you don't have to do anything or recreate anything. We will upgrade the production bits in our service.

The fix is currently being deployed to our canary clusters.

[danielnelson]

@arerlend Do you know when we can expect the fix to be generally available? I still see the response did not contain all the expected topic/partition blocks error unless I have the workaround from #1542.

[arerlend]

Sorry I missed this. Deployment had to be put on hold for the holidays but is resuming now. If anyone needs specific namespaces to be patched, feel free to reach out to me via email.

[arerlend]

The deployment was completed for all clusters, but subsequently had to be rolled back on a few clusters, as there was an unrelated regression that affected a few customers. Please ping me directly with your namespace if you are still seeing this issue.

[elsesiy]

@arerlend Did this also apply to dedicated instances? Our deployment stopped working with the following error at exactly Jan, 6th 9:00pm PST:

{"error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)","id":"39a4f074-c0f2-4664-991a-769f6ed28db6","level":"fatal","message":"couldn't establish connection to kafka","ts":1578595531249}

I tried re-creating consumergroups but nothing works anymore. Is this related?

[arerlend]

@elsesiy dedicated instances got the same upgrade. Can you open a new issue against our Github page? That way you can fill out the checklist and I'll have the info I need.
https://github.com/Azure/azure-event-hubs-for-kafka

[arerlend]

@faec this fix was deployed again last week. Issue should be resolved.

[probot-shopify[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the master branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.