Pulsar: standalone mode cannot sync MaxMessageSize to bookkeeper

[freeznet]

Describe the bug
after upgrade to v2.4, a new setting MaxMessageSize was introduced to make messagesize more than 5MB, but bookkeeper still using 5M as nettyMaxFrameSizeBytes, which should be sync with MaxMessageSize

Additional context

io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5242880: 18405407 - discarded
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:522) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:500) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:387) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:430) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:343) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

[jiazhai]

@freeznet , I would like to confirm the config for this issue reproduce:

• pulsar set MaxMessageSize to a value > 5MB
• bookkeeper set nettyMaxFrameSizeBytes to 5MB

Then we will meet this issue when use pulsar?

If this is the situation, currently, user have to manually do the size settings between pulsar and bookkeeper.
@zymap for more details

[freeznet]

@jiazhai i see, but i havnt find any docs about how to manually change nettyMaxFrameSizeBytes in standalone mode, need more info.

[sijie]

@jiazhai @zymap : there are a couple things we can do here:

1) we need to document the end-to-end steps for adjusting the max message size. I think the part how to configure bookies is missed.

2) we need to set the nettyMaxFrameSizeBytes bookkeeper client used to the value of maxMessageSize configured at the broker. I am not sure if we did it now or not. If we didn't, we should do it.

[zymap]

@freeznet You can try to add nettyMaxFrameSizeBytes=5242880 in standalone.conf directly.

[zymap]

@sijie

1. we need to document the end-to-end steps for adjusting the max message size. I think the part how to configure bookies is missed.

Yeah. I will add a doc to introduce how to config bookies.

1. we need to set the nettyMaxFrameSizeBytes bookkeeper client used to the value of maxMessageSize configured at the broker. I am not sure if we did it now or not. If we didn't, we should do it.

yes, it is.

[freeznet]

@freeznet You can try to add nettyMaxFrameSizeBytes=5242880 in standalone.conf directly.

@zymap problem solved, thanks!

[Lanayx]

The issue is still relevant with 2.5.0 - you still need to set nettyMaxFrameSizeBytes to bookeeper.conf and not rely on maxMessageSize for broker

[sijie]

@Lanayx - standalone loads nettyMaxFrameSizeBytes from standalone.conf for bookkeeper as well. So you don't need to set nettyMaxFrameSizeBytes in bookkeeper.conf. But if you deploy a cluster, you need to set nettyMaxFrameSizeBytes in both broker.conf (to be used in bookkeeper client) and bookkeeper.conf (to be used in bookie).