Confluent-kafka-dotnet: Getting Confluent.Kafka.KafkaException: Broker: Message size too large

[xs2bharat]

Description

I followed multiple links online for sending large message to kafka but still getting this error.

exception:
Confluent.Kafka.KafkaException: Broker: Message size too large
at Confluent.Kafka.Producer.Produce(String topic, Byte[] val, Int32 valOffset, Int32 valLength, Byte[] key, Int32 keyOffset, Int32 keyLength, Nullable1 timestamp, Int32 partition, Boolean blockIfQueueFull, IDeliveryHandler deliveryHandler) at Confluent.Kafka.SerializingProducer2.Produce(String topic, TKey key, TValue val, Nullable1 timestamp, Int32 partition, Boolean blockIfQueueFull, IDeliveryHandler2 deliveryHandler)
at Confluent.Kafka.SerializingProducer2.ProduceAsync(String topic, TKey key, TValue val, IDeliveryHandler2 deliveryHandler)
at Confluent.Kafka.Producer2.ProduceAsync(String topic, TKey key, TValue val, IDeliveryHandler2 deliveryHandler)

Kafka broker settings:

message.max.bytes=10000000
replica.fetch.max.bytes=10485760
fetch.message.max.bytes=10000000

Message size is 1.1 mb

Not sure if i am missing any settings.

How to reproduce

Checklist

Please provide the following information:

• [ ] Confluent.Kafka nuget version: 0.11.4
• [ ] Apache Kafka version: 0.11.0.1
• [ ] Client configuration:
• [ ] Operating system:
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[xs2bharat]

To solve above issues, i was trying to add "max.request.size" into producer config but for some reason, confluent packages was not able to instantiate producer client. But adding "message.max.bytes" to producer client did solved the above error, which is weird because this setting is already configured on broker.

[mhowlett]

interesting. since the exception corresponds to librdkafka reporting a broker error, i'm surprised the solution was to increase message.max.bytes on the client. i want to investigate this further (todo).

note that client side and server side configuration are completely separate.

[mhowlett]

i'm closing this, since I believe it's almost certainly due to a mis-configuration.

[DHirani]

I am getting the same error, how was this resolved?

[Caskia]

@DHirani
You need to set the message.max.bytes at broker side

[wangjia184]

            var config = new Dictionary<string, object>
            {
                { "bootstrap.servers", AppConfig.Kafka.Brokers },
                { "message.max.bytes", 1024*1024*10 }, // https://github.com/confluentinc/confluent-kafka-dotnet/issues/532
            };
            log.DebugFormat("KafkaProducer connecting to `{0}`", config["bootstrap.servers"]);
            _producer = new Producer<byte[], byte[]>(config, new ByteArraySerializer(), new ByteArraySerializer());

[michael-elly]

I have the same issue... I have increased the message.max.bytes and replica.fetch.max.bytes on the broker side. But I don't see what I need to configure on the producer config on Confluent.Net. There is no message.max.size property (or anything similar to it) on the ProducerConfig class object.