最近碰到一个问题,某个业务的同学说他们服务使用的某个 topic 的某个 Queue 发生了位点回退。
我看了下确实是回退了,然后看了下日志,是初始化 client 时请求 offset timeout,然后 请求的 offset 变成了 -1。
broker 在接收到 -1 的请求时,返回了 minLogicOffset ,然后 client 带着 minLogicOffset 把之前 broker 保存的正常的位点覆盖了。
想请教下为什么当拿着 offset = -1 来请求时返回的是 minLogicOffset,而不是返回 broker 保存的该 group 该 Queue 的位点呢?
client日志
2021-03-02 15:15:12 WARN PullMessageService - invokeSync: wait response timeout exception, the channel[brokerip:10911]
2021-03-02 15:15:12 WARN NettyClientWorkerThread_2 - receive response, but not matched any request, brokerip:10911
2021-03-02 15:15:12 WARN NettyClientWorkerThread_2 - RemotingCommand [code=0, language=JAVA, version=317, opaque=6774, flag(B)=1, remark=null, extFields={offset=5757922}, serializeTypeCurrentRPC=JSON]
2021-03-02 15:15:12 WARN PullMessageService - fetchConsumeOffsetFromBroker exception, MessageQueue [topic=tp_inner_order_operation, brokerName=broker-06, queueId=10]
2021-03-02 15:15:12 WARN NettyClientPublicExecutor_3 - the pull request offset illegal, PullRequest [consumerGroup=ts_biz_operation_group, messageQueue=MessageQueue [topic=tp_inner_order_operation, brokerName=broker-0
6, queueId=10], nextOffset=-1] PullResult [pullStatus=OFFSET_ILLEGAL, nextBeginOffset=5531762, minOffset=5531762, maxOffset=5757977, msgFoundList=0]
2021-03-02 15:15:22 WARN PullMessageServiceScheduledThread - unlock messageQueue. group:ts_biz_operation_group, clientId:clientip@30487, mq:MessageQueue [topic=tp_inner_order_operation, brokerName=broker-06, queue
Id=10]
2021-03-02 15:15:22 WARN PullMessageServiceScheduledThread - fix the pull request offset, PullRequest [consumerGroup=ts_biz_operation_group, messageQueue=MessageQueue [topic=tp_inner_order_operation, brokerName=broker
-06, queueId=10], nextOffset=5531762]
2021-03-02 15:15:45 WARN PullMessageService - the consumer message buffer is full, so do flow control, minOffset=5531803, maxOffset=5532809, size=1008, pullRequest=PullRequest [consumerGroup=ts_biz_operation_group, me
ssageQueue=MessageQueue [topic=tp_inner_order_operation, brokerName=broker-06, queueId=10], nextOffset=5532810], flowControlTimes=1
broker日志
broker.log:2021-03-02 15:15:12 INFO PullMessageThread_5 - the request offset too small. group=ts_biz_operation_group, topic=tp_inner_order_operation, requestOffset=-1, brokerMinOffset=5531762, clientIp=/clientip:24944
broker.log:2021-03-02 15:15:12 WARN PullMessageThread_5 - PULL_OFFSET_MOVED:correction offset. topic=tp_inner_order_operation, groupId=ts_biz_operation_group, requestOffset=-1, newOffset=5531762, suggestBrokerId=0
broker.log:2021-03-02 15:15:22 WARN ConsumerManageThread_15 - [NOTIFYME]update consumer offset less than store. clientHost=clientip:24944, key=tp_inner_order_operation@ts_biz_operation_group, queueId=10, requestOffset=5531762, storeOffset=5757922
store.log:2021-03-02 04:54:08 INFO StoreScheduledThread1 - Compute logical min offset: 5531762, topic: tp_inner_order_operation, queueId: 10
这个可以算是一个BUG,具体表现为多条队列只有一部分位点变成了 Broker 最小位点(minOffset),在网络不稳定时会获取不到位点,而 SDK 没有区分原因到底是 Timeout 还是获取不到,修复方案是 SDK 区分这几种异常
这个可以算是一个BUG,具体表现为多条队列只有一部分位点变成了 Broker 最小位点(minOffset),在网络不稳定时会获取不到位点,而 SDK 没有区分原因到底是 Timeout 还是获取不到,修复方案是 SDK 区分这几种异常
我感觉这块也是个BUG,但是我看最新版本一直没修复就有点困惑
Most helpful comment
这个可以算是一个BUG,具体表现为多条队列只有一部分位点变成了 Broker 最小位点(minOffset),在网络不稳定时会获取不到位点,而 SDK 没有区分原因到底是 Timeout 还是获取不到,修复方案是 SDK 区分这几种异常