Confluent-kafka-go: Crash on message retry enqueue
Description
We have large scale deployment of kafka producers using confluent-kafka-go and come across many occurrences of crashes associated with the following assertion failure:rd_kafka_assert(NULL, rkmq->rkmq_msg_cnt > 0) in rdkafka_msg.h. It appears to occur more often while we have under replicated partitions or while we perform partition reassignment.
The following stack trace has been found to be shared among all the crash occurrences:
```[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program terminated with signal SIGABRT, Aborted.
0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
+------------------------------------------------------------------------------+
| backtrace |
+------------------------------------------------------------------------------+
0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
1 0x0000000000458d82 in runtime.raisebadsignal (c=0x7f0fb77fbd98, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
2 0x00000000004591a3 in runtime.badsignal (c=0x7f0fb77fbd98, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
3 0x0000000000458988 in runtime.sigtrampgo (ctx=0x7f0fb77fbe40, info=0x7f0fb77fbf70, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
4 0x0000000000471863 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
5
6 0x00007f0fdaecdc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
7 0x00007f0fdaed1028 in __GI_abort () at abort.c:89
8 0x000000000066456a in rd_kafka_crash (file=file@entry=0x7c15c3 "rdkafka_msg.h", line=line@entry=263, function=function@entry=0x7c1980 <__FUNCTION__.19041> "rd_kafka_msgq_deq", rk=rk@entry=0x0, reason=reason@entry=0x7c1708 "assert: rkmq->rkmq_msg_cnt > 0") at rdkafka.c:3102
9 0x000000000041a520 in rd_kafka_msgq_deq (do_count=, rkm=, rkmq=) at rdkafka_msg.h:263
10 0x0000000000683e09 in rd_kafka_msgq_deq (do_count=, rkm=, rkmq=) at rdkafka_msg.h:264
11 rd_kafka_msgq_age_scan (rkmq=0x1ba88, rkmq@entry=0x7f0ebc044a88, timedout=0x1bad5, timedout@entry=0x7f0fb77fc5f0, now=6, now@entry=2173403318391) at rdkafka_msg.c:577
12 0x0000000000687875 in rd_kafka_topic_scan_all (rk=rk@entry=0x7f0fcc001400, now=2173403318391) at rdkafka_topic.c:1135
13 0x0000000000662e56 in rd_kafka_topic_scan_tmr_cb (rkts=, arg=) at rdkafka.c:1194
14 0x000000000068a448 in rd_kafka_timers_run (rkts=rkts@entry=0x7f0fcc001c18, timeout_us=timeout_us@entry=0) at rdkafka_timer.c:251
15 0x000000000066c8d7 in rd_kafka_thread_main (arg=arg@entry=0x7f0fcc001400) at rdkafka.c:1270
16 0x00000000006cdb97 in _thrd_wrapper_function (aArg=) at tinycthread.c:624
17 0x00007f0fdb470184 in start_thread (arg=0x7f0fb77fe700) at pthread_create.c:312
18 0x00007f0fdaf94ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
How to reproduce
================
We tried and have not yet been successful reproducing in a lab environment. We'd like to know of any suggestions of changes in the librdkafka code such that it prints out things useful to diagnose this issue. Also, we briefly reviewed the later librdkafka commits and didn't find anything that looks like it addresses this crash, but if we misidentified this, we would be happy to rebuild and retest with the lastest librdkafka commit in master.
Checklist
=========
Please provide the following information:
- [x] confluent-kafka-go and librdkafka version (`LibraryVersion()`):
confluent-kafka-go commit: `1112e2c08a15bce669a99cc35284a690fd2086b8`
librdkafka: `0.11.4-RC1B`
- [x] Apache Kafka broker version: 0.11.0.1
- [x] Client configuration: `ConfigMap{...}`
"bootstrap.servers": brokerList,
"queue.buffering.max.ms": "500",
"compression.codec": "gzip",
"batch.num.messages": "256",
"security.protocol": "ssl",
"queue.buffering.max.messages": 8000,
"broker.version.fallback": "0.10.2.1",
"statistics.interval.ms": 60000,
```
- [x] Operating system: ubuntu 14.04
- [ ] Provide client logs (with
"debug": ".."as necessary)
We didn't know witch debug setting would be useful for this so didn't enable any debug flag. Please let us know if there is a debug flag we should enable. - [ ] Provide broker log excerpts
- [x] Critical issue
lintang0
👍1
All 39 comments
Hmm, interesting.
One thing that would be interesting is reproducing this with debug=broker,topic,metadata enabled and providing us with the logs.
edenhill
on 13 Jun 2018
@edenhill we shall try that, thanks! Hopefully it doesn't fill up our disks (or disk bandwidth)... :)
billygout
on 13 Jun 2018
assert.log
@edenhill attach the producer log when assertion happened with debug=broker,topic,metadata Hope you can find what's going on... thx.
lintang0
on 18 Jun 2018
Thank you, will look into it!
edenhill
on 18 Jun 2018
+------------------------------------------------------------------------------+
| core dump backtrace |
+------------------------------------------------------------------------------+
0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
1 0x0000000000458d82 in runtime.raisebadsignal (c=0x7f56ccff6918, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
2 0x00000000004591a3 in runtime.badsignal (c=0x7f56ccff6918, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
3 0x0000000000458988 in runtime.sigtrampgo (ctx=0x7f56ccff69c0, info=0x7f56ccff6af0, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
4 0x0000000000471863 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
5
6 0x00007f57a35bdc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
7 0x00007f57a35c1028 in __GI_abort () at abort.c:89
8 0x000000000066382a in rd_kafka_crash (file=file@entry=0x7c07a3 "rdkafka_msg.h", line=line@entry=263, function=function@entry=0x7cdd50 <__FUNCTION__.19041> "rd_kafka_msgq_deq", rk=rk@entry=0x0, reason=reason@entry=0x7c08e8 "assert: rkmq->rkmq_msg_cnt > 0") at rdkafka.c:3102
9 0x00000000006d8e70 in rd_kafka_msgq_deq (do_count=1, rkm=0x7f55ac01e0f0, rkmq=0x7f574c047838) at rdkafka_msg.h:263
10 rd_kafka_msgset_writer_write_msgq (rkmq=0x7f574c047838, msetw=0x7f56ccff71e0) at rdkafka_msgset_writer.c:757
11 rd_kafka_msgset_create_ProduceRequest (rkb=rkb@entry=0x7f574c00cfa0, rktp=rktp@entry=0x7f574c047740, MessageSetSizep=MessageSetSizep@entry=0x7f56ccff73d8) at rdkafka_msgset_writer.c:1223
12 0x00000000006aa38b in rd_kafka_ProduceRequest (rkb=rkb@entry=0x7f574c00cfa0, rktp=rktp@entry=0x7f574c047740) at rdkafka_request.c:1958
13 0x0000000000681b13 in rd_kafka_toppar_producer_serve (do_timeout_scan=1, next_wakeup=, now=4641297975967, rktp=0x7f574c047740, rkb=0x7f574c00cfa0) at rdkafka_broker.c:2488
14 rd_kafka_broker_produce_toppars (do_timeout_scan=1, next_wakeup=, now=4641297975967, rkb=0x7f574c00cfa0) at rdkafka_broker.c:2526
15 rd_kafka_broker_producer_serve (rkb=0x7f574c00cfa0) at rdkafka_broker.c:2571
16 rd_kafka_broker_thread_main (arg=arg@entry=0x7f574c00cfa0) at rdkafka_broker.c:3309
17 0x00000000006cce57 in _thrd_wrapper_function (aArg=) at tinycthread.c:624
18 0x00007f57a3b60184 in start_thread (arg=0x7f56ccff9700) at pthread_create.c:312
19 0x00007f57a3684ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
lintang0
on 18 Jun 2018
Thank you for providing a backtrace and a log! ❤️
edenhill
on 18 Jun 2018
@edenhill Currently, the librdkafka was built without -DNDEBUG flag. I'm wonder what may happen if the DNDEBUG in the librdkafka build? How does the librdkafka handle the same situation w/wo the flag? Is there a recommend way to use librdkafka for the flag? What happens if we don't abort at that point. will it cause more corruption, or not.thanks!
lintang0
on 21 Jun 2018
librdkafka itself does not build differently with NDEBUG, but it is possible that included system header files might.
There is an --enable-devel option that enables some extra assertions, which you could try.
I've analysed your log and backtraces, and while I definitively think that something is wrong I haven't been able to identify exactly what. Also tried reproducing with lousy connections without success.
edenhill
on 24 Jun 2018
@edenhill what are the chances that --enable-devel will reveal assertions that will help track down this bug? Unfortunately, we can't reproduce this either in the lab, so we would have to enable this in production, so we need to make sure we don't cause more crashes, that wouldn't have occurred otherwise.
billygout
on 29 Jun 2018
There are about 50 devel-asserts:
$ git grep dassert | wc -l
50
And they should not introduce any additional assert-crashes that don't actually show a problem.
There's quite a bit of dasserts in the queue code, so I do think it might help us find the issue
edenhill
on 30 Jun 2018
@edenhill we've set --enable-devel, and I'm seeing some crash stack traces I haven't noticed before, so I'll report them here even though I'm not sure whether they are related to this issue.
In all the below, we are using this commit, which includes the "connection on demand" code: https://github.com/edenhill/librdkafka/commit/ed23da66ed21dac4787f50d73afaaee531e97889
There are 3 different stack traces below. Please let me know if there's any other info I can provide.
stack trace 1
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000006c039c in rd_kafka_msgq_insert_msgq (
destq=destq@entry=0x7fbac02463f8, srcq=0x7fbaacfdea80,
cmp=0x681f20 ) at rdkafka_partition.c:733
+------------------------------------------------------------------------------+
| backtrace |
+------------------------------------------------------------------------------+
#0 0x00000000006c039c in rd_kafka_msgq_insert_msgq (destq=destq@entry=0x7fbac02463f8, srcq=0x7fbaacfdea80, cmp=0x681f20 ) at rdkafka_partition.c:733
#1 0x00000000006c05bc in rd_kafka_retry_msgq (destq=destq@entry=0x7fbac02463f8, srcq=srcq@entry=0x7fb910041588, incr_retry=incr_retry@entry=0, max_retries=, backoff=backoff@entry=7928373100364, cmp=) at rdkafka_partition.c:784
#2 0x00000000006c071d in rd_kafka_toppar_retry_msgq (rktp=rktp@entry=0x7fbac0246300, rkmq=rkmq@entry=0x7fb910041588, incr_retry=0) at rdkafka_partition.c:807
#3 0x00000000006a3266 in rd_kafka_handle_Produce (rk=, rkb=, err=RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION, reply=, request=0x7fb910041400, opaque=0x7fbac0246300) at rdkafka_request.c:1883
#4 0x000000000069262e in rd_kafka_buf_callback (rk=, rkb=rkb@entry=0x7fbac0220e50, err=err@entry=RD_KAFKA_RESP_ERR_NO_ERROR, response=response@entry=0x7fb910041210, request=request@entry=0x7fb910041400) at rdkafka_buf.c:450
#5 0x000000000067669c in rd_kafka_req_response (rkbuf=0x7fb910041210, rkb=0x7fbac0220e50) at rdkafka_broker.c:1340
#6 rd_kafka_recv (rkb=rkb@entry=0x7fbac0220e50) at rdkafka_broker.c:1452
#7 0x000000000068ffd3 in rd_kafka_transport_io_event (rktrans=0x7fb9100404c0, events=5) at rdkafka_transport.c:1419
#8 0x0000000000690c9e in rd_kafka_transport_io_serve (rktrans=, timeout_ms=) at rdkafka_transport.c:1467
#9 0x000000000067d357 in rd_kafka_broker_serve (rkb=rkb@entry=0x7fbac0220e50, abs_timeout=abs_timeout@entry=7928374000285) at rdkafka_broker.c:2593
#10 0x000000000067e083 in rd_kafka_broker_producer_serve (rkb=0x7fbac0220e50) at rdkafka_broker.c:2882
#11 rd_kafka_broker_thread_main (arg=arg@entry=0x7fbac0220e50) at rdkafka_broker.c:3611
#12 0x00000000006d0007 in _thrd_wrapper_function (aArg=) at tinycthread.c:624
#13 0x00007fbb1cd73184 in start_thread (arg=0x7fbaacfe1700) at pthread_create.c:312
#14 0x00007fbb1c897ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
here are local vars in frame #0, some of which were created by this macro https://github.com/edenhill/librdkafka/blob/4e7a46701ecce7297b2298885da980be7856e5f9/src/rdsysqueue.h#L248:
local vars in frame 0
(gdb) info locals
_elm = 0x7fba240ca1d0
_last = 0x7fba080ba420
_aft = 0x0
at = 0x7fba4c03afc0
first = 0x7fba240ca1d0
dest_first = <optimized out>
__PRETTY_FUNCTION__ = "rd_kafka_msgq_insert_msgq"
stack trace 2
Program terminated with signal SIGSEGV, Segmentation fault.
#0 rd_kafka_msgset_writer_write_msg_headers (msetw=,
hdrs=0x7f699c03dd60) at rdkafka_msgset_writer.c:469
+------------------------------------------------------------------------------+
| backtrace |
+------------------------------------------------------------------------------+
#0 rd_kafka_msgset_writer_write_msg_headers (msetw=, hdrs=0x7f699c03dd60) at rdkafka_msgset_writer.c:469
#1 rd_kafka_msgset_writer_write_msg_v2 (msetw=, rkm=, Offset=, MsgAttributes=, free_cb=0x0) at rdkafka_msgset_writer.c:659
#2 0x00000000006d9ebf in rd_kafka_msgset_writer_write_msg (msetw=msetw@entry=0x7f6a61ff1220, rkm=rkm@entry=0x7f699c03d200, Offset=Offset@entry=61, MsgAttributes=MsgAttributes@entry=0 '\000', free_cb=free_cb@entry=0x0) at rdkafka_msgset_writer.c:692
#3 0x00000000006db3fe in rd_kafka_msgset_writer_write_msgq (rkmq=0x7f6a68252918, msetw=0x7f6a61ff1220) at rdkafka_msgset_writer.c:774
#4 rd_kafka_msgset_create_ProduceRequest (rkb=rkb@entry=0x7f6a68170f70, rktp=rktp@entry=0x7f6a68252820, MessageSetSizep=MessageSetSizep@entry=0x7f6a61ff1418) at rdkafka_msgset_writer.c:1235
#5 0x00000000006ac6eb in rd_kafka_ProduceRequest (rkb=rkb@entry=0x7f6a68170f70, rktp=rktp@entry=0x7f6a68252820) at rdkafka_request.c:1958
#6 0x000000000067e183 in rd_kafka_toppar_producer_serve (do_timeout_scan=0, next_wakeup=, now=6640202889974, rktp=0x7f6a68252820, rkb=0x7f6a68170f70) at rdkafka_broker.c:2790
#7 rd_kafka_broker_produce_toppars (do_timeout_scan=0, next_wakeup=, now=6640202889974, rkb=0x7f6a68170f70) at rdkafka_broker.c:2828
#8 rd_kafka_broker_producer_serve (rkb=0x7f6a68170f70) at rdkafka_broker.c:2873
#9 rd_kafka_broker_thread_main (arg=arg@entry=0x7f6a68170f70) at rdkafka_broker.c:3611
#10 0x00000000006d0007 in _thrd_wrapper_function (aArg=) at tinycthread.c:624
#11 0x00007f6af1313184 in start_thread (arg=0x7f6a61ff3700) at pthread_create.c:312
#12 0x00007f6af0e37ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
local vars in frames 0 and 1
(gdb) frame
#0 rd_kafka_msgset_writer_write_msg_headers (msetw=<optimized out>, hdrs=0x7f699c03dd60)
at rdkafka_msgset_writer.c:469
469 in rdkafka_msgset_writer.c
(gdb) info locals
i = 2
written = <optimized out>
rkbuf = 0x7f694c019130
hdr = 0x20
(gdb) frame 1
#1 rd_kafka_msgset_writer_write_msg_v2 (msetw=<optimized out>, rkm=<optimized out>,
Offset=<optimized out>, MsgAttributes=<optimized out>, free_cb=0x0) at rdkafka_msgset_writer.c:659
659 in rdkafka_msgset_writer.c
(gdb) info locals
rkbuf = <optimized out>
MessageSize = <optimized out>
varint_Length = "\242\020\065\223n"
varint_TimestampDelta = "\262\fj\177\000\000\210\020\377a"
varint_OffsetDelta = "z\177\000\000\f\000\000\000\000"
varint_KeyLen = ":\000\000\000"
varint_ValueLen = "\250\017\001Li"
varint_HeaderCount = "\324\375\003\200\020"
HeaderCount = <optimized out>
HeaderSize = <optimized out>
(gdb)
stack trace 3
Program terminated with signal SIGABRT, Aborted.
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
+------------------------------------------------------------------------------+
| backtrace |
+------------------------------------------------------------------------------+
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#1 0x0000000000459372 in runtime.raisebadsignal (c=0x7f6cb2fda718, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
#2 0x0000000000459793 in runtime.badsignal (c=0x7f6cb2fda718, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
#3 0x0000000000458f78 in runtime.sigtrampgo (ctx=0x7f6cb2fda7c0, info=0x7f6cb2fda8f0, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
#4 0x0000000000471e53 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
#5
#6 0x00007f6d47556c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#7 0x00007f6d4755a028 in __GI_abort () at abort.c:89
#8 0x00007f6d4754fbf6 in __assert_fail_base (fmt=0x7f6d476a4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7dc84f "p", file=file@entry=0x7cc781 "rd.h", line=line@entry=101, function=function@entry=0x7e31f0 <__PRETTY_FUNCTION__.5801> "rd_malloc") at assert.c:92
#9 0x00007f6d4754fca2 in __GI___assert_fail (assertion=assertion@entry=0x7dc84f "p", file=file@entry=0x7cc781 "rd.h", line=line@entry=101, function=function@entry=0x7e31f0 <__PRETTY_FUNCTION__.5801> "rd_malloc") at assert.c:101
#10 0x000000000041b576 in rd_malloc (sz=280208498437018) at rd.h:101
#11 0x00000000006e8e5a in rd_malloc (sz=) at rdbuf.c:144
#12 rd_buf_alloc_segment0 (rbuf=rbuf@entry=0x7f6b50003448, size=280208498437018) at rdbuf.c:132
#13 0x00000000006e8e79 in rd_buf_alloc_segment (rbuf=rbuf@entry=0x7f6b50003448, min_size=, max_size=) at rdbuf.c:165
#14 0x00000000006e90e3 in rd_buf_write_ensure (rbuf=rbuf@entry=0x7f6b50003448, min_size=min_size@entry=140104249228592, max_size=max_size@entry=0) at rdbuf.c:206
#15 0x00000000006e929a in rd_buf_write (rbuf=rbuf@entry=0x7f6b50003448, payload=payload@entry=0x7f6c9001c8a0, size=size@entry=140104249228592) at rdbuf.c:447
#16 0x00000000006da57d in rd_kafka_buf_write (len=140104249228592, data=0x7f6c9001c8a0, rkbuf=0x7f6b50003420) at rdkafka_buf.h:712
#17 rd_kafka_msgset_writer_write_msg_headers (msetw=, hdrs=0x7f6c9001b810) at rdkafka_msgset_writer.c:470
#18 rd_kafka_msgset_writer_write_msg_v2 (msetw=, rkm=, Offset=, MsgAttributes=, free_cb=0x0) at rdkafka_msgset_writer.c:659
#19 0x00000000006d9ebf in rd_kafka_msgset_writer_write_msg (msetw=msetw@entry=0x7f6cb2fdb220, rkm=rkm@entry=0x7f6c9000b530, Offset=Offset@entry=104, MsgAttributes=MsgAttributes@entry=0 '\000', free_cb=free_cb@entry=0x0) at rdkafka_msgset_writer.c:692
#20 0x00000000006db3fe in rd_kafka_msgset_writer_write_msgq (rkmq=0x7f6cc424e678, msetw=0x7f6cb2fdb220) at rdkafka_msgset_writer.c:774
#21 rd_kafka_msgset_create_ProduceRequest (rkb=rkb@entry=0x7f6cc41dfb30, rktp=rktp@entry=0x7f6cc424e580, MessageSetSizep=MessageSetSizep@entry=0x7f6cb2fdb418) at rdkafka_msgset_writer.c:1235
#22 0x00000000006ac6eb in rd_kafka_ProduceRequest (rkb=rkb@entry=0x7f6cc41dfb30, rktp=rktp@entry=0x7f6cc424e580) at rdkafka_request.c:1958
#23 0x000000000067e183 in rd_kafka_toppar_producer_serve (do_timeout_scan=0, next_wakeup=, now=3766428500291, rktp=0x7f6cc424e580, rkb=0x7f6cc41dfb30) at rdkafka_broker.c:2790
#24 rd_kafka_broker_produce_toppars (do_timeout_scan=0, next_wakeup=, now=3766428500291, rkb=0x7f6cc41dfb30) at rdkafka_broker.c:2828
#25 rd_kafka_broker_producer_serve (rkb=0x7f6cc41dfb30) at rdkafka_broker.c:2873
#26 rd_kafka_broker_thread_main (arg=arg@entry=0x7f6cc41dfb30) at rdkafka_broker.c:3611
#27 0x00000000006d0007 in _thrd_wrapper_function (aArg=) at tinycthread.c:624
#28 0x00007f6d47af9184 in start_thread (arg=0x7f6cb2fdd700) at pthread_create.c:312
#29 0x00007f6d4761dffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
local vars in frames 15, 17, 18
(gdb) frame 15
#15 0x00000000006e929a in rd_buf_write (rbuf=rbuf@entry=0x7f6b50003448,
payload=payload@entry=0x7f6c9001c8a0, size=size@entry=140104249228592) at rdbuf.c:447
447 in rdbuf.c
(gdb) info locals
remains = 140104249228592
initial_absof = 92098
psrc = 0x7f6c9001c8a0 " "
__PRETTY_FUNCTION__ = "rd_buf_write"
(gdb) frame 17
#17 rd_kafka_msgset_writer_write_msg_headers (msetw=<optimized out>, hdrs=0x7f6c9001b810)
at rdkafka_msgset_writer.c:470
470 rdkafka_msgset_writer.c: No such file or directory.
(gdb) info locals
i = 0
written = <optimized out>
rkbuf = 0x7f6b50003420
hdr = 0x7f6c9001c880
(gdb) frame 18
#18 rd_kafka_msgset_writer_write_msg_v2 (msetw=<optimized out>, rkm=<optimized out>,
Offset=<optimized out>, MsgAttributes=<optimized out>, free_cb=0x0) at rdkafka_msgset_writer.c:659
659 in rdkafka_msgset_writer.c
(gdb) info locals
rkbuf = <optimized out>
MessageSize = <optimized out>
varint_Length = "\204\t5\223n"
varint_TimestampDelta = "\264#l\177\000\000\210\260", <incomplete sequence \375\262>
varint_OffsetDelta = "\320\001\000\000\f\000\000\000\000"
varint_KeyLen = "0\000\000\000"
varint_ValueLen = "\234\b\000Pk"
varint_HeaderCount = "\330\375\003\200\260"
HeaderCount = <optimized out>
HeaderSize = <optimized out>
(gdb)
Thanks!
Corey
billygout
on 25 Jul 2018
Thank you for the great information.
Did you ever see this problems on a proper release which did not have the sparse connection patches in?
edenhill
on 7 Aug 2018
Hi @edenhill I searched all our core dumps for incidents of stack traces 1, 2, and 3 (detailed above) which occurred before we included the sparse connection patches, and came up with the following results:
found stack trace 1
found stack trace 3
did NOT find stack trace 2 (but it is pretty rare even now, so we can't really conclude much from it not having been found before the patches)
billygout
on 8 Aug 2018
Thank you. What scale are we talking about here?
How many clients are running and how many crashes do you get in a week, day, ..?
edenhill
on 8 Aug 2018
@edenhill number of clients is around 15,000.
i'll give the crash frequency per stack trace:
stack trace 3: very rare
stack trace 1: about 1.6 per day on average (on various days: as many as 6 and as little as 0)
stack trace 2: very rare
And for the stack trace reported at the top of this issue (https://github.com/confluentinc/confluent-kafka-go/issues/203#issue-332163655), the frqeuency is: about 2 per day
billygout
on 10 Aug 2018
That is a lot of clients :)
I'm not really sure what the problem is, and the relative rarity of the crashes doesn't really help either to narrow it down.
What could help is building librdkafka with --disable-optimization --enable-devel to get better stack traces when it crashes, the optimization currently removes too much info.
Would that be possible for you to try?
#15 0x00000000006e929a in rd_buf_write (rbuf=rbuf@entry=0x7f6b50003448,
payload=payload@entry=0x7f6c9001c8a0, size=size@entry=140104249228592) at rdbuf.c:447
The size variable value is 0x7f6c9001bd30 in hex, which looks very much like a heap pointer. This could either be optimizations playing tricks with the backtrace, or a stack issue.
edenhill
on 13 Aug 2018
@edenhill yes, we can try it. Since we're in production, we'll have watch out for slowdowns which could impact us, so it would be nice if there were a patch we could apply to capture the missing info only where it's needed, rather than turning off all optimizations, but if you don't think that's gonna do it, we'll just try what you suggest.
Also, if for any stack traces, you'd like to see any other gdb command output, let me know
billygout
on 13 Aug 2018
Sorry, I don't have any specific ideas what this could be at this point in time, so I think non-optimized gdb stack traces is the next step.
I wouldn't be so worried about performance, the drop isn't huge with -O0
edenhill
on 13 Aug 2018
Ack, we've begun the process. Hopefully in a week or so, we'll have the
stack traces (and logs if you need them).
On Mon, Aug 13, 2018, 12:21 PM Magnus Edenhill notifications@github.com
wrote:
Sorry, I don't have any specific ideas what this could be at this point in
time, so I think non-optimized gdb stack traces is the next step.
I wouldn't be so worried about performance, the drop isn't huge with -O0—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/confluent-kafka-go/issues/203#issuecomment-412633594,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABoBEkSZUMXJ0agYyBL7cUdlsq6gjLF4ks5uQdHOgaJpZM4Um7tI
.
billygout
on 13 Aug 2018
👍1
Thank you!
edenhill
on 13 Aug 2018
@edenhill I'm happy to report that we have 3 new stack traces that have been produced after we deployed with --disable-optimizations.
They are as follows:
Stack trace 4
Program terminated with signal SIGABRT, Aborted.
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#1 0x0000000000458ee2 in runtime.raisebadsignal (c=0x7fc272fd0098, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
#2 0x0000000000459303 in runtime.badsignal (c=0x7fc272fd0098, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
#3 0x0000000000458ae8 in runtime.sigtrampgo (ctx=0x7fc272fd0140, info=0x7fc272fd0270, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
#4 0x00000000004719c3 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
#5
#6 0x00007fc308f48c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#7 0x00007fc308f4c028 in __GI_abort () at abort.c:89
#8 0x000000000066d24b in rd_kafka_crash (file=0x805a28 "rdkafka_msgset_writer.c", line=734, function=0x806400 <__FUNCTION__.21745> "rd_kafka_msgset_writer_write_msgq", rk=0x0, reason=0x805b48 "assert: rkm") at rdkafka.c:3370
#9 0x00000000006f59c7 in rd_kafka_msgset_writer_write_msgq (msetw=0x7fc272fd0990, rkmq=0x7fc29024f258) at rdkafka_msgset_writer.c:734
#10 0x00000000006f711a in rd_kafka_msgset_create_ProduceRequest (rkb=0x7fc290215010, rktp=0x7fc29024f160, MessageSetSizep=0x7fc272fd0a58) at rdkafka_msgset_writer.c:1235
#11 0x00000000006b58cf in rd_kafka_ProduceRequest (rkb=0x7fc290215010, rktp=0x7fc29024f160) at rdkafka_request.c:1958
#12 0x000000000067d76b in rd_kafka_toppar_producer_serve (rkb=0x7fc290215010, rktp=0x7fc29024f160, now=6005758677507, next_wakeup=0x7fc272fd0bf8, do_timeout_scan=0) at rdkafka_broker.c:2790
#13 0x000000000067d859 in rd_kafka_broker_produce_toppars (rkb=0x7fc290215010, now=6005758677507, next_wakeup=0x7fc272fd0c50, do_timeout_scan=0) at rdkafka_broker.c:2828
#14 0x000000000067d9c6 in rd_kafka_broker_producer_serve (rkb=0x7fc290215010) at rdkafka_broker.c:2873
#15 0x00000000006832c1 in rd_kafka_broker_thread_main (arg=0x7fc290215010) at rdkafka_broker.c:3611
#16 0x00000000006dda20 in _thrd_wrapper_function (aArg=0x7fc290014ff0) at tinycthread.c:624
#17 0x00007fc3094eb184 in start_thread (arg=0x7fc272fd5700) at pthread_create.c:312
#18 0x00007fc30900fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Stack trace 4 local variables
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
No locals.
#1 0x0000000000458ee2 in runtime.raisebadsignal (c=0x7fc272fd0098, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
handler = 0
#2 0x0000000000459303 in runtime.badsignal (c=0x7fc272fd0098, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
No locals.
#3 0x0000000000458ae8 in runtime.sigtrampgo (ctx=0x7fc272fd0140, info=0x7fc272fd0270, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
g = 0x646441203a313833
gsignalStack = {
stack = {
lo = 7738980122663795308,
hi = 7020958085903691873
},
stackguard0 = 8314035674280125742,
stackguard1 = 8387231026747040628,
stktopsp = 3834585504778565946
}
setStack = 53
sp = 8319028371727136819
st = {
ss_sp = 0x0,
ss_flags = 0,
pad_cgo_0 = "\000\000\000",
ss_size = 0
}
st = {
ss_sp = 0x0,
ss_flags = 0,
pad_cgo_0 = "\000\000\000",
ss_size = 0
}
#4 0x00000000004719c3 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
No locals.
#5
No locals.
#6 0x00007fc308f48c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 43940
selftid = 44195
#7 0x00007fc308f4c028 in __GI_abort () at abort.c:89
save_stage = 2
act = {
__sigaction_handler = {
sa_handler = 0x7fc272fd59c0,
sa_sigaction = 0x7fc272fd59c0
},
sa_mask = {
__val = {140473129588480, 140473129568336, 140475667769079, 140462610448389, 0, 140466301705648, 140475645398320, 140473618723168, 140473129568336, 0, 140475667797669, 83, 140475646350253, 0, 0, 140473129558016}
},
sa_flags = 1929205504,
sa_restorer = 0x775f726574697277
}
sigs = {
__val = {32, 0 }
}
#8 0x000000000066d24b in rd_kafka_crash (file=0x805a28 "rdkafka_msgset_writer.c", line=734, function=0x806400 <__FUNCTION__.21745> "rd_kafka_msgset_writer_write_msgq", rk=0x0, reason=0x805b48 "assert: rkm") at rdkafka.c:3370
No locals.
#9 0x00000000006f59c7 in rd_kafka_msgset_writer_write_msgq (msetw=0x7fc272fd0990, rkmq=0x7fc29024f258) at rdkafka_msgset_writer.c:734
rkbuf = 0x7fc0dc041bc0
rktp = 0x7fc29024f160
rkb = 0x7fc290215010
len = 114
max_msg_size = 1000000
int_latency_base = 6006061503145
MaxTimestamp = 0
rkm = 0x0
msgcnt = 0
now = 6005761503145
__FUNCTION__ = "rd_kafka_msgset_writer_write_msgq"
__PRETTY_FUNCTION__ = "rd_kafka_msgset_writer_write_msgq"
#10 0x00000000006f711a in rd_kafka_msgset_create_ProduceRequest (rkb=0x7fc290215010, rktp=0x7fc29024f160, MessageSetSizep=0x7fc272fd0a58) at rdkafka_msgset_writer.c:1235
msetw = {
msetw_rkbuf = 0x7fc0dc041bc0,
msetw_ApiVersion = 3,
msetw_MsgVersion = 2,
msetw_features = 512,
msetw_msgcntmax = 1,
msetw_messages_len = 0,
msetw_messages_kvlen = 0,
msetw_MessageSetSize = 61,
msetw_of_MessageSetSize = 49,
msetw_of_start = 53,
msetw_relative_offsets = 1,
msetw_Attributes = 0,
msetw_MaxTimestamp = 0,
msetw_of_CRC = 70,
msetw_firstmsg = {
of = 114,
timestamp = 0
},
msetw_rkb = 0x7fc290215010,
msetw_rktp = 0x7fc29024f160
}
#11 0x00000000006b58cf in rd_kafka_ProduceRequest (rkb=0x7fc290215010, rktp=0x7fc29024f160) at rdkafka_request.c:1958
rkbuf = 0x7fc0dc0419d0
rkt = 0x7fc2bc0008e0
MessageSetSize = 0
cnt = 96
now = 6005761503138
first_msg_timeout = 297107
tmout = 297107
__PRETTY_FUNCTION__ = "rd_kafka_ProduceRequest"
#12 0x000000000067d76b in rd_kafka_toppar_producer_serve (rkb=0x7fc290215010, rktp=0x7fc29024f160, now=6005758677507, next_wakeup=0x7fc272fd0bf8, do_timeout_scan=0) at rdkafka_broker.c:2790
cnt = 7776
r = 96
rkm = 0x7fc2c00872f0
move_cnt = 1877
__PRETTY_FUNCTION__ = "rd_kafka_toppar_producer_serve"
#13 0x000000000067d859 in rd_kafka_broker_produce_toppars (rkb=0x7fc290215010, now=6005758677507, next_wakeup=0x7fc272fd0c50, do_timeout_scan=0) at rdkafka_broker.c:2828
this_next_wakeup = 6005759677507
rktp = 0x7fc29024f160
cnt = 0
ret_next_wakeup = 6005759677507
#14 0x000000000067d9c6 in rd_kafka_broker_producer_serve (rkb=0x7fc290215010) at rdkafka_broker.c:2873
now = 6005758677507
next_wakeup = 6005759677507
do_timeout_scan = 0
timeout_scan = {
ri_ts_last = 6005758200042,
ri_fixed = 0,
ri_backoff = 0
}
__FUNCTION__ = "rd_kafka_broker_producer_serve"
#15 0x00000000006832c1 in rd_kafka_broker_thread_main (arg=0x7fc290215010) at rdkafka_broker.c:3611
backoff = 39179821383
rkb = 0x7fc290215010
rk = 0x7fc2e001e440
#16 0x00000000006dda20 in _thrd_wrapper_function (aArg=0x7fc290014ff0) at tinycthread.c:624
fun = 0x682db9
arg = 0x7fc290215010
res = 0
ti = 0x7fc290014ff0
#17 0x00007fc3094eb184 in start_thread (arg=0x7fc272fd5700) at pthread_create.c:312
__res =
pd = 0x7fc272fd5700
now =
unwind_buf = {
cancel_jmp_buf = {{
jmp_buf = {140473129588480, -7734031682798833756, 0, 0, 140473129589184, 140473129588480, 7723628089257275300, 7722865993587011492},
mask_was_saved = 0
}},
priv = {
pad = {0x0, 0x0, 0x0, 0x0},
data = {
prev = 0x0,
cleanup = 0x0,
canceltype = 0
}
}
}
not_first_call =
pagesize_m1 =
sp =
freesize =
__PRETTY_FUNCTION__ = "start_thread"
#18 0x00007fc30900fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
Stack trace 5
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000006cc9e7 in rd_kafka_msgq_insert_msgq (destq=0x7f95a424ed18, srcq=0x7f9584fc4170, cmp=0x68bb70 ) at rdkafka_partition.c:733
#1 0x00000000006ccb49 in rd_kafka_retry_msgq (destq=0x7f95a424ed18, srcq=0x7f93e8058568, incr_retry=0, max_retries=2, backoff=8963038600256, cmp=0x68bb70 ) at rdkafka_partition.c:784
#2 0x00000000006ccbe4 in rd_kafka_toppar_retry_msgq (rktp=0x7f95a424ec20, rkmq=0x7f93e8058568, incr_retry=0) at rdkafka_partition.c:807
#3 0x00000000006b56ab in rd_kafka_handle_Produce (rk=0x7f95e001e440, rkb=0x7f95a4100490, err=RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION, reply=0x7f93e80581f0, request=0x7f93e80583e0, opaque=0x7f95a424ec20) at rdkafka_request.c:1883
#4 0x00000000006a094c in rd_kafka_buf_callback (rk=0x7f95e001e440, rkb=0x7f95a4100490, err=RD_KAFKA_RESP_ERR_NO_ERROR, response=0x7f93e80581f0, request=0x7f93e80583e0) at rdkafka_buf.c:450
#5 0x00000000006781a7 in rd_kafka_req_response (rkb=0x7f95a4100490, rkbuf=0x7f93e80581f0) at rdkafka_broker.c:1340
#6 0x0000000000678b69 in rd_kafka_recv (rkb=0x7f95a4100490) at rdkafka_broker.c:1452
#7 0x000000000069d10d in rd_kafka_transport_io_event (rktrans=0x7f93e80576a0, events=1) at rdkafka_transport.c:1419
#8 0x000000000069d28b in rd_kafka_transport_io_serve (rktrans=0x7f93e80576a0, timeout_ms=999) at rdkafka_transport.c:1467
#9 0x000000000067d144 in rd_kafka_broker_serve (rkb=0x7f95a4100490, abs_timeout=8963039500140) at rdkafka_broker.c:2593
#10 0x000000000067da48 in rd_kafka_broker_producer_serve (rkb=0x7f95a4100490) at rdkafka_broker.c:2882
#11 0x00000000006832c1 in rd_kafka_broker_thread_main (arg=0x7f95a4100490) at rdkafka_broker.c:3611
#12 0x00000000006dda20 in _thrd_wrapper_function (aArg=0x7f95a4014ff0) at tinycthread.c:624
#13 0x00007f960f54e184 in start_thread (arg=0x7f9584fc9700) at pthread_create.c:312
#14 0x00007f960f072ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Stack trace 5 local variables
#0 0x00000000006cc9e7 in rd_kafka_msgq_insert_msgq (destq=0x7f95a424ed18, srcq=0x7f9584fc4170, cmp=0x68bb70 ) at rdkafka_partition.c:733
_elm = 0x7f94ec010670
_last = 0x7f951804fb30
_aft = 0x0
at = 0x7f950402f500
first = 0x7f94ec010670
dest_first = 0x7f950402f500
__PRETTY_FUNCTION__ = "rd_kafka_msgq_insert_msgq"
#1 0x00000000006ccb49 in rd_kafka_retry_msgq (destq=0x7f95a424ed18, srcq=0x7f93e8058568, incr_retry=0, max_retries=2, backoff=8963038600256, cmp=0x68bb70 ) at rdkafka_partition.c:784
retryable = {
rkmq_msgs = {
tqh_first = 0x7f94ec010670,
tqh_last = 0x7f951804fb78
},
rkmq_msg_cnt = 85,
rkmq_msg_bytes = 121805
}
rkm = 0x0
tmp = 0x0
#2 0x00000000006ccbe4 in rd_kafka_toppar_retry_msgq (rktp=0x7f95a424ec20, rkmq=0x7f93e8058568, incr_retry=0) at rdkafka_partition.c:807
rk = 0x7f95e001e440
backoff = 8963038600256
r = 32661
#3 0x00000000006b56ab in rd_kafka_handle_Produce (rk=0x7f95e001e440, rkb=0x7f95a4100490, err=RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION, reply=0x7f93e80581f0, request=0x7f93e80583e0, opaque=0x7f95a424ec20) at rdkafka_request.c:1883
incr_retry = 0
actions = 4
actstr = "pw\005\350\223\177\000\000p", '\000' , "\026\027\003\003\000.\177\000\000\005\000\000\000\000\000\000\000\000_\216\307\025\366\322\304\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000"
s_rktp = 0x7f95a424ec20
rktp = 0x7f95a424ec20
offset = -1
timestamp = -1
#4 0x00000000006a094c in rd_kafka_buf_callback (rk=0x7f95e001e440, rkb=0x7f95a4100490, err=RD_KAFKA_RESP_ERR_NO_ERROR, response=0x7f93e80581f0, request=0x7f93e80583e0) at rdkafka_buf.c:450
__FUNCTION__ = "rd_kafka_buf_callback"
#5 0x00000000006781a7 in rd_kafka_req_response (rkb=0x7f95a4100490, rkbuf=0x7f93e80581f0) at rdkafka_broker.c:1340
req = 0x7f93e80583e0
__FUNCTION__ = "rd_kafka_req_response"
__PRETTY_FUNCTION__ = "rd_kafka_req_response"
#6 0x0000000000678b69 in rd_kafka_recv (rkb=0x7f95a4100490) at rdkafka_broker.c:1452
rkbuf = 0x7f93e80581f0
r = 42
errstr = "Protocol parse failure", '\000'
err = RD_KAFKA_RESP_ERR_NO_ERROR
log_decode_errors = 3
__PRETTY_FUNCTION__ = "rd_kafka_recv"
__FUNCTION__ = "rd_kafka_recv"
#7 0x000000000069d10d in rd_kafka_transport_io_event (rktrans=0x7f93e80576a0, events=1) at rdkafka_transport.c:1419
errstr = "\001\001\374\204\225\177\000\000\362\320m\000\000\000\000\000\000X(\244\225\177\000\000\000\000\000\000\000\000\000\000\255ug\000\000\000\000\000M\327m\000\000\000\000\000\360I\374\204\000\000\000\000\360\344\001\340\225\177\000\000\360I\374\204\225\177\000\000\241vg", '\000' , "@\344\001\340\225\177\000\000\260K\374\204\225\177\000\000\223\315g\000\000\000\000\000\202\017\000\374\225\177\000\000\205\000\000\000\006\000\000\000(\003\000\000\225\177\000\000\360Y\374\204\225\177\000\000 \354$\244\225\177\000\000\271\237l\000\000\000\000\000\370\354$\244\225\177\000\000\030\355$\244\225\177\000\000\000\000\000\000\000\000\000\000\220\004\020\244\225\177\000\000\240J\374\204\001\000\000\000"...
r = 0
rkb = 0x7f95a4100490
__FUNCTION__ = "rd_kafka_transport_io_event"
#8 0x000000000069d28b in rd_kafka_transport_io_serve (rktrans=0x7f93e80576a0, timeout_ms=999) at rdkafka_transport.c:1467
rkb = 0x7f95a4100490
events = 1
#9 0x000000000067d144 in rd_kafka_broker_serve (rkb=0x7f95a4100490, abs_timeout=8963039500140) at rdkafka_broker.c:2593
blocking_max_ms = 999
now = 140280679368392
initial_state = 4
remains_ms = 999
#10 0x000000000067da48 in rd_kafka_broker_producer_serve (rkb=0x7f95a4100490) at rdkafka_broker.c:2882
now = 8963038500140
next_wakeup = 8963039500140
do_timeout_scan = 0
timeout_scan = {
ri_ts_last = 8963038365620,
ri_fixed = 0,
ri_backoff = 0
}
__FUNCTION__ = "rd_kafka_broker_producer_serve"
#11 0x00000000006832c1 in rd_kafka_broker_thread_main (arg=0x7f95a4100490) at rdkafka_broker.c:3611
backoff = 33599565714
rkb = 0x7f95a4100490
rk = 0x7f95e001e440
#12 0x00000000006dda20 in _thrd_wrapper_function (aArg=0x7f95a4014ff0) at tinycthread.c:624
fun = 0x682db9
arg = 0x7f95a4100490
res = 0
ti = 0x7f95a4014ff0
#13 0x00007f960f54e184 in start_thread (arg=0x7f9584fc9700) at pthread_create.c:312
__res =
pd = 0x7f9584fc9700
now =
unwind_buf = {
cancel_jmp_buf = {{
jmp_buf = {140280158000896, -6903502394537161093, 0, 0, 140280158001600, 140280158000896, 6853032631719546491, 6853858026208087675},
mask_was_saved = 0
}},
priv = {
pad = {0x0, 0x0, 0x0, 0x0},
data = {
prev = 0x0,
cleanup = 0x0,
canceltype = 0
}
}
}
not_first_call =
pagesize_m1 =
sp =
freesize =
__PRETTY_FUNCTION__ = "start_thread"
#14 0x00007f960f072ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
Stack trace 6
Program terminated with signal SIGABRT, Aborted.
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#1 0x0000000000458ee2 in runtime.raisebadsignal (c=0x7f00eefb8058, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
#2 0x0000000000459303 in runtime.badsignal (c=0x7f00eefb8058, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
#3 0x0000000000458ae8 in runtime.sigtrampgo (ctx=0x7f00eefb8100, info=0x7f00eefb8230, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
#4 0x00000000004719c3 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
#5
#6 0x00007f019c28ec37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#7 0x00007f019c292028 in __GI_abort () at abort.c:89
#8 0x000000000066d24b in rd_kafka_crash (file=0x805884 "rdkafka_msg.h", line=265, function=0x806470 <__FUNCTION__.18362> "rd_kafka_msgq_deq", rk=0x0, reason=0x8058e0 "assert: rkmq->rkmq_msg_bytes >= (int64_t)(rkm->rkm_len+rkm->rkm_key_len)") at rdkafka.c:3370
#9 0x00000000006f3d96 in rd_kafka_msgq_deq (rkmq=0x7f013023f658, rkm=0x7f0194132650, do_count=1) at rdkafka_msg.h:264
#10 0x00000000006f5b54 in rd_kafka_msgset_writer_write_msgq (msetw=0x7f00eefb8990, rkmq=0x7f013023f658) at rdkafka_msgset_writer.c:760
#11 0x00000000006f711a in rd_kafka_msgset_create_ProduceRequest (rkb=0x7f01301cfd30, rktp=0x7f013023f560, MessageSetSizep=0x7f00eefb8a58) at rdkafka_msgset_writer.c:1235
#12 0x00000000006b58cf in rd_kafka_ProduceRequest (rkb=0x7f01301cfd30, rktp=0x7f013023f560) at rdkafka_request.c:1958
#13 0x000000000067d76b in rd_kafka_toppar_producer_serve (rkb=0x7f01301cfd30, rktp=0x7f013023f560, now=8578348201907, next_wakeup=0x7f00eefb8bf8, do_timeout_scan=1) at rdkafka_broker.c:2790
#14 0x000000000067d859 in rd_kafka_broker_produce_toppars (rkb=0x7f01301cfd30, now=8578348201907, next_wakeup=0x7f00eefb8c50, do_timeout_scan=1) at rdkafka_broker.c:2828
#15 0x000000000067d9c6 in rd_kafka_broker_producer_serve (rkb=0x7f01301cfd30) at rdkafka_broker.c:2873
#16 0x00000000006832c1 in rd_kafka_broker_thread_main (arg=0x7f01301cfd30) at rdkafka_broker.c:3611
#17 0x00000000006dda20 in _thrd_wrapper_function (aArg=0x7f0130014e30) at tinycthread.c:624
#18 0x00007f019c831184 in start_thread (arg=0x7f00eefbd700) at pthread_create.c:312
#19 0x00007f019c355ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Stack trace 6 local variables
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
No locals.
#1 0x0000000000458ee2 in runtime.raisebadsignal (c=0x7f00eefb8058, sig=6) at /usr/local/go/src/runtime/signal_unix.go:491
handler = 0
#2 0x0000000000459303 in runtime.badsignal (c=0x7f00eefb8058, sig=6) at /usr/local/go/src/runtime/signal_unix.go:600
No locals.
#3 0x0000000000458ae8 in runtime.sigtrampgo (ctx=0x7f00eefb8100, info=0x7f00eefb8230, sig=6) at /usr/local/go/src/runtime/signal_unix.go:297
g = 0x74656e2e6e646374
gsignalStack = {
stack = {
lo = 139641986189376,
hi = 8319057139774026843
},
stackguard0 = 7738980122663795308,
stackguard1 = 7020958085903626337,
stktopsp = 8314035674280125742
}
setStack = false
sp = 139641986189616
st = {
ss_sp = 0x0,
ss_flags = 0,
pad_cgo_0 = "\000\000\000",
ss_size = 0
}
st = {
ss_sp = 0x0,
ss_flags = 0,
pad_cgo_0 = "\000\000\000",
ss_size = 0
}
#4 0x00000000004719c3 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
No locals.
#5
No locals.
#6 0x00007f019c28ec37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 157312
selftid = 157748
#7 0x00007f019c292028 in __GI_abort () at abort.c:89
save_stage = 2
act = {
__sigaction_handler = {
sa_handler = 0x7f00eefbd9c0,
sa_sigaction = 0x7f00eefbd9c0
},
sa_mask = {
__val = {139641986209536, 139641986189344, 139644913796855, 5, 0, 139637105103752, 139644891426096, 139643079357792, 139641986189344, 0, 139644913825445, 118, 139644892378029, 0, 0, 139641986179024}
},
sa_flags = -285485312,
sa_restorer = 0x722b6e656c5f6d6b
}
sigs = {
__val = {32, 0 }
}
#8 0x000000000066d24b in rd_kafka_crash (file=0x805884 "rdkafka_msg.h", line=265, function=0x806470 <__FUNCTION__.18362> "rd_kafka_msgq_deq", rk=0x0, reason=0x8058e0 "assert: rkmq->rkmq_msg_bytes >= (int64_t)(rkm->rkm_len+rkm->rkm_key_len)") at rdkafka.c:3370
No locals.
#9 0x00000000006f3d96 in rd_kafka_msgq_deq (rkmq=0x7f013023f658, rkm=0x7f0194132650, do_count=1) at rdkafka_msg.h:264
__FUNCTION__ = "rd_kafka_msgq_deq"
#10 0x00000000006f5b54 in rd_kafka_msgset_writer_write_msgq (msetw=0x7f00eefb8990, rkmq=0x7f013023f658) at rdkafka_msgset_writer.c:760
rkbuf = 0x7effcc0c2a10
rktp = 0x7f013023f560
rkb = 0x7f01301cfd30
len = 245889
max_msg_size = 1000000
int_latency_base = 8578648201923
MaxTimestamp = 1534547296979
rkm = 0x7f0194132650
msgcnt = 193
now = 8578348201923
__FUNCTION__ = "rd_kafka_msgset_writer_write_msgq"
__PRETTY_FUNCTION__ = "rd_kafka_msgset_writer_write_msgq"
#11 0x00000000006f711a in rd_kafka_msgset_create_ProduceRequest (rkb=0x7f01301cfd30, rktp=0x7f013023f560, MessageSetSizep=0x7f00eefb8a58) at rdkafka_msgset_writer.c:1235
msetw = {
msetw_rkbuf = 0x7effcc0c2a10,
msetw_ApiVersion = 3,
msetw_MsgVersion = 2,
msetw_features = 512,
msetw_msgcntmax = 256,
msetw_messages_len = 0,
msetw_messages_kvlen = 239467,
msetw_MessageSetSize = 61,
msetw_of_MessageSetSize = 49,
msetw_of_start = 53,
msetw_relative_offsets = 1,
msetw_Attributes = 0,
msetw_MaxTimestamp = 0,
msetw_of_CRC = 70,
msetw_firstmsg = {
of = 114,
timestamp = 1534547296974
},
msetw_rkb = 0x7f01301cfd30,
msetw_rktp = 0x7f013023f560
}
#12 0x00000000006b58cf in rd_kafka_ProduceRequest (rkb=0x7f01301cfd30, rktp=0x7f013023f560) at rdkafka_request.c:1958
rkbuf = 0x7f013023f658
rkt = 0x7f0184037850
MessageSetSize = 0
cnt = 32512
now = 139644755660800
first_msg_timeout = 0
tmout = -285504864
__PRETTY_FUNCTION__ = "rd_kafka_ProduceRequest"
#13 0x000000000067d76b in rd_kafka_toppar_producer_serve (rkb=0x7f01301cfd30, rktp=0x7f013023f560, now=8578348201907, next_wakeup=0x7f00eefb8bf8, do_timeout_scan=1) at rdkafka_broker.c:2790
cnt = 0
r = 930
rkm = 0x7f01940e5000
move_cnt = 930
__PRETTY_FUNCTION__ = "rd_kafka_toppar_producer_serve"
#14 0x000000000067d859 in rd_kafka_broker_produce_toppars (rkb=0x7f01301cfd30, now=8578348201907, next_wakeup=0x7f00eefb8c50, do_timeout_scan=1) at rdkafka_broker.c:2828
this_next_wakeup = 8578349201907
rktp = 0x7f013023f560
cnt = 0
ret_next_wakeup = 8578349201907
#15 0x000000000067d9c6 in rd_kafka_broker_producer_serve (rkb=0x7f01301cfd30) at rdkafka_broker.c:2873
now = 8578348201907
next_wakeup = 8578349201907
do_timeout_scan = 1
timeout_scan = {
ri_ts_last = 8578348201907,
ri_fixed = 0,
ri_backoff = 0
}
__FUNCTION__ = "rd_kafka_broker_producer_serve"
#16 0x00000000006832c1 in rd_kafka_broker_thread_main (arg=0x7f01301cfd30) at rdkafka_broker.c:3611
backoff = 17280618007
rkb = 0x7f01301cfd30
rk = 0x7f0194001510
#17 0x00000000006dda20 in _thrd_wrapper_function (aArg=0x7f0130014e30) at tinycthread.c:624
fun = 0x682db9
arg = 0x7f01301cfd30
res = 0
ti = 0x7f0130014e30
#18 0x00007f019c831184 in start_thread (arg=0x7f00eefbd700) at pthread_create.c:312
__res =
pd = 0x7f00eefbd700
now =
unwind_buf = {
cancel_jmp_buf = {{
jmp_buf = {139641986209536, -4311259190233080936, 0, 0, 139641986210240, 139641986209536, 4191313503858252696, 4190719978189544344},
mask_was_saved = 0
}},
priv = {
pad = {0x0, 0x0, 0x0, 0x0},
data = {
prev = 0x0,
cleanup = 0x0,
canceltype = 0
}
}
}
not_first_call =
pagesize_m1 =
sp =
freesize =
__PRETTY_FUNCTION__ = "start_thread"
#19 0x00007f019c355ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
Awesome, thank you!
I think I've found the (or a) issue:
when a produce request fails with a retriable error the messages from the request
are put back on the partition queue in the original order and the produce request is re-created.
If multiple subsequent requests fail in short succession, such as when the partition leader changes which is the case in at least some of these crashes, the messages from the request are simply appended to the end of the queue. And this is where the bug lies:
https://github.com/edenhill/librdkafka/blob/master/src/rdsysqueue.h#L251
The destination queue (i.e., rktp_xmit_msgq) tailq header's tqh_last is not updated to point at the new last element, so the queue will be corrupt.
I'll make a fix soon.
I'm very impressed with the amount of information you've provided for this issue. :heart:
edenhill
on 21 Aug 2018
❤1
@edenhill can't tell you how happy we are to hear that! Thanks!
billygout
on 21 Aug 2018
@billygout Bad news, I've been trying to provoke the issue locally based on the above theory, but to no avail. I don't think the code in question (TAILQ_INSERT_TAIL) is buggy, and have had the code re-reviewed to make sure.
But I'm quite sure the bug is related to the partition queue, so I'll try other means of reproducing the problem.
Thanks for your patience, back to the drawing board.
edenhill
on 22 Aug 2018
Do you still have the corefile for Stack trace 5 local variables above?
If so, could you do the following for me, in frame 0:
#0 0x00000000006cc9e7 in rd_kafka_msgq_insert_msgq (destq=0x7f95a424ed18, srcq=0x7f9584fc4170, cmp=0x68bb70 ) at rdkafka_partition.c:733
_elm = 0x7f94ec010670
_last = 0x7f951804fb30
_aft = 0x0
at = 0x7f950402f500
first = 0x7f94ec010670
dest_first = 0x7f950402f500
__PRETTY_FUNCTION__ = "rd_kafka_msgq_insert_msgq"
gdb commands:
disassemble
p *at
p *destq
p destq.rkmq_msgs
p *first
p *dest_first
@edenhill sure. let me know if any other commands would help. here's what you asked for:
(gdb) frame 0
#0 0x00000000006cc9e7 in rd_kafka_msgq_insert_msgq (destq=0x7f95a424ed18, srcq=0x7f9584fc4170, cmp=0x68bb70 <rd_kafka_msg_cmp_msgseq>)
at rdkafka_partition.c:733
733 in rdkafka_partition.c
(gdb) disassemble
Dump of assembler code for function rd_kafka_msgq_insert_msgq:
0x00000000006cc7e7 <+0>: push %rbp
0x00000000006cc7e8 <+1>: mov %rsp,%rbp
0x00000000006cc7eb <+4>: sub $0x50,%rsp
0x00000000006cc7ef <+8>: mov %rdi,-0x38(%rbp)
0x00000000006cc7f3 <+12>: mov %rsi,-0x40(%rbp)
0x00000000006cc7f7 <+16>: mov %rdx,-0x48(%rbp)
0x00000000006cc7fb <+20>: mov -0x40(%rbp),%rax
0x00000000006cc7ff <+24>: mov (%rax),%rax
0x00000000006cc802 <+27>: mov %rax,-0x30(%rbp)
0x00000000006cc806 <+31>: cmpq $0x0,-0x30(%rbp)
0x00000000006cc80b <+36>: sete %al
0x00000000006cc80e <+39>: movzbl %al,%eax
0x00000000006cc811 <+42>: test %rax,%rax
0x00000000006cc814 <+45>: je 0x6cc81b <rd_kafka_msgq_insert_msgq+52>
0x00000000006cc816 <+47>: jmpq 0x6cca40 <rd_kafka_msgq_insert_msgq+601>
0x00000000006cc81b <+52>: mov -0x38(%rbp),%rax
0x00000000006cc81f <+56>: mov (%rax),%rax
0x00000000006cc822 <+59>: mov %rax,-0x28(%rbp)
0x00000000006cc826 <+63>: cmpq $0x0,-0x28(%rbp)
0x00000000006cc82b <+68>: sete %al
0x00000000006cc82e <+71>: movzbl %al,%eax
0x00000000006cc831 <+74>: test %rax,%rax
0x00000000006cc834 <+77>: je 0x6cc84e <rd_kafka_msgq_insert_msgq+103>
0x00000000006cc836 <+79>: mov -0x40(%rbp),%rdx
0x00000000006cc83a <+83>: mov -0x38(%rbp),%rax
0x00000000006cc83e <+87>: mov %rdx,%rsi
0x00000000006cc841 <+90>: mov %rax,%rdi
0x00000000006cc844 <+93>: callq 0x6c9fbb <rd_kafka_msgq_move>
0x00000000006cc849 <+98>: jmpq 0x6cca40 <rd_kafka_msgq_insert_msgq+601>
0x00000000006cc84e <+103>: mov -0x28(%rbp),%rcx
0x00000000006cc852 <+107>: mov -0x30(%rbp),%rdx
0x00000000006cc856 <+111>: mov -0x48(%rbp),%rax
0x00000000006cc85a <+115>: mov %rcx,%rsi
0x00000000006cc85d <+118>: mov %rdx,%rdi
0x00000000006cc860 <+121>: callq *%rax
0x00000000006cc862 <+123>: test %eax,%eax
0x00000000006cc864 <+125>: jns 0x6cc891 <rd_kafka_msgq_insert_msgq+170>
0x00000000006cc866 <+127>: mov -0x38(%rbp),%rdx
0x00000000006cc86a <+131>: mov -0x40(%rbp),%rax
0x00000000006cc86e <+135>: mov %rdx,%rsi
0x00000000006cc871 <+138>: mov %rax,%rdi
0x00000000006cc874 <+141>: callq 0x6c9f13 <rd_kafka_msgq_concat>
0x00000000006cc879 <+146>: mov -0x40(%rbp),%rdx
0x00000000006cc87d <+150>: mov -0x38(%rbp),%rax
---Type <return> to continue, or q <return> to quit---
0x00000000006cc881 <+154>: mov %rdx,%rsi
0x00000000006cc884 <+157>: mov %rax,%rdi
0x00000000006cc887 <+160>: callq 0x6c9fbb <rd_kafka_msgq_move>
0x00000000006cc88c <+165>: jmpq 0x6cca40 <rd_kafka_msgq_insert_msgq+601>
0x00000000006cc891 <+170>: mov -0x38(%rbp),%rax
0x00000000006cc895 <+174>: mov 0x8(%rax),%rax
0x00000000006cc899 <+178>: mov 0x8(%rax),%rax
0x00000000006cc89d <+182>: mov (%rax),%rcx
0x00000000006cc8a0 <+185>: mov -0x30(%rbp),%rdx
0x00000000006cc8a4 <+189>: mov -0x48(%rbp),%rax
0x00000000006cc8a8 <+193>: mov %rcx,%rsi
0x00000000006cc8ab <+196>: mov %rdx,%rdi
0x00000000006cc8ae <+199>: callq *%rax
0x00000000006cc8b0 <+201>: test %eax,%eax
0x00000000006cc8b2 <+203>: jle 0x6cc8cc <rd_kafka_msgq_insert_msgq+229>
0x00000000006cc8b4 <+205>: mov -0x40(%rbp),%rdx
0x00000000006cc8b8 <+209>: mov -0x38(%rbp),%rax
0x00000000006cc8bc <+213>: mov %rdx,%rsi
0x00000000006cc8bf <+216>: mov %rax,%rdi
0x00000000006cc8c2 <+219>: callq 0x6c9f13 <rd_kafka_msgq_concat>
0x00000000006cc8c7 <+224>: jmpq 0x6cca40 <rd_kafka_msgq_insert_msgq+601>
0x00000000006cc8cc <+229>: mov -0x48(%rbp),%rdx
0x00000000006cc8d0 <+233>: mov -0x30(%rbp),%rcx
0x00000000006cc8d4 <+237>: mov -0x38(%rbp),%rax
0x00000000006cc8d8 <+241>: mov %rcx,%rsi
0x00000000006cc8db <+244>: mov %rax,%rdi
0x00000000006cc8de <+247>: callq 0x688a7b <rd_kafka_msgq_find_pos>
0x00000000006cc8e3 <+252>: mov %rax,-0x20(%rbp)
0x00000000006cc8e7 <+256>: cmpq $0x0,-0x20(%rbp)
0x00000000006cc8ec <+261>: je 0x6cc8f7 <rd_kafka_msgq_insert_msgq+272>
0x00000000006cc8ee <+263>: mov $0x42,%eax
0x00000000006cc8f3 <+268>: test %al,%al
0x00000000006cc8f5 <+270>: jne 0x6cc916 <rd_kafka_msgq_insert_msgq+303>
0x00000000006cc8f7 <+272>: lea 0x1334e2(%rip),%rcx # 0x7ffde0 <__PRETTY_FUNCTION__.21898>
0x00000000006cc8fe <+279>: mov $0x2d7,%edx
0x00000000006cc903 <+284>: lea 0x132174(%rip),%rsi # 0x7fea7e
0x00000000006cc90a <+291>: lea 0x13235f(%rip),%rdi # 0x7fec70
0x00000000006cc911 <+298>: callq 0x415f70 <__assert_fail@plt>
0x00000000006cc916 <+303>: mov -0x38(%rbp),%rax
0x00000000006cc91a <+307>: mov 0x8(%rax),%rax
0x00000000006cc91e <+311>: mov 0x8(%rax),%rax
0x00000000006cc922 <+315>: mov (%rax),%rax
0x00000000006cc925 <+318>: cmp -0x20(%rbp),%rax
0x00000000006cc929 <+322>: jne 0x6cc989 <rd_kafka_msgq_insert_msgq+418>
0x00000000006cc92b <+324>: mov -0x40(%rbp),%rax
---Type <return> to continue, or q <return> to quit---
0x00000000006cc92f <+328>: mov (%rax),%rax
0x00000000006cc932 <+331>: test %rax,%rax
0x00000000006cc935 <+334>: je 0x6cca02 <rd_kafka_msgq_insert_msgq+539>
0x00000000006cc93b <+340>: mov -0x38(%rbp),%rax
0x00000000006cc93f <+344>: mov 0x8(%rax),%rax
0x00000000006cc943 <+348>: mov -0x40(%rbp),%rdx
0x00000000006cc947 <+352>: mov (%rdx),%rdx
0x00000000006cc94a <+355>: mov %rdx,(%rax)
0x00000000006cc94d <+358>: mov -0x40(%rbp),%rax
0x00000000006cc951 <+362>: mov (%rax),%rax
0x00000000006cc954 <+365>: mov -0x38(%rbp),%rdx
0x00000000006cc958 <+369>: mov 0x8(%rdx),%rdx
0x00000000006cc95c <+373>: mov %rdx,0x50(%rax)
0x00000000006cc960 <+377>: mov -0x40(%rbp),%rax
0x00000000006cc964 <+381>: mov 0x8(%rax),%rdx
0x00000000006cc968 <+385>: mov -0x38(%rbp),%rax
0x00000000006cc96c <+389>: mov %rdx,0x8(%rax)
0x00000000006cc970 <+393>: mov -0x40(%rbp),%rax
0x00000000006cc974 <+397>: movq $0x0,(%rax)
0x00000000006cc97b <+404>: mov -0x40(%rbp),%rdx
0x00000000006cc97f <+408>: mov -0x40(%rbp),%rax
0x00000000006cc983 <+412>: mov %rdx,0x8(%rax)
0x00000000006cc987 <+416>: jmp 0x6cca02 <rd_kafka_msgq_insert_msgq+539>
0x00000000006cc989 <+418>: mov -0x40(%rbp),%rax
0x00000000006cc98d <+422>: mov (%rax),%rax
0x00000000006cc990 <+425>: mov %rax,-0x18(%rbp)
0x00000000006cc994 <+429>: mov -0x40(%rbp),%rax
0x00000000006cc998 <+433>: mov 0x8(%rax),%rax
0x00000000006cc99c <+437>: mov 0x8(%rax),%rax
0x00000000006cc9a0 <+441>: mov (%rax),%rax
0x00000000006cc9a3 <+444>: mov %rax,-0x10(%rbp)
0x00000000006cc9a7 <+448>: mov -0x20(%rbp),%rax
0x00000000006cc9ab <+452>: mov 0x48(%rax),%rax
0x00000000006cc9af <+456>: mov %rax,-0x8(%rbp)
0x00000000006cc9b3 <+460>: mov -0x20(%rbp),%rax
0x00000000006cc9b7 <+464>: mov -0x18(%rbp),%rdx
0x00000000006cc9bb <+468>: mov %rdx,0x48(%rax)
0x00000000006cc9bf <+472>: mov -0x20(%rbp),%rax
0x00000000006cc9c3 <+476>: lea 0x48(%rax),%rdx
0x00000000006cc9c7 <+480>: mov -0x18(%rbp),%rax
0x00000000006cc9cb <+484>: mov %rdx,0x50(%rax)
0x00000000006cc9cf <+488>: mov -0x10(%rbp),%rax
0x00000000006cc9d3 <+492>: mov -0x8(%rbp),%rdx
0x00000000006cc9d7 <+496>: mov %rdx,0x48(%rax)
0x00000000006cc9db <+500>: mov -0x10(%rbp),%rax
---Type <return> to continue, or q <return> to quit---
0x00000000006cc9df <+504>: lea 0x48(%rax),%rdx
0x00000000006cc9e3 <+508>: mov -0x8(%rbp),%rax
=> 0x00000000006cc9e7 <+512>: mov %rdx,0x50(%rax)
0x00000000006cc9eb <+516>: mov -0x40(%rbp),%rax
0x00000000006cc9ef <+520>: movq $0x0,(%rax)
0x00000000006cc9f6 <+527>: mov -0x40(%rbp),%rdx
0x00000000006cc9fa <+531>: mov -0x40(%rbp),%rax
0x00000000006cc9fe <+535>: mov %rdx,0x8(%rax)
0x00000000006cca02 <+539>: mov -0x38(%rbp),%rax
0x00000000006cca06 <+543>: mov 0x10(%rax),%edx
0x00000000006cca09 <+546>: mov -0x40(%rbp),%rax
0x00000000006cca0d <+550>: mov 0x10(%rax),%eax
0x00000000006cca10 <+553>: add %eax,%edx
0x00000000006cca12 <+555>: mov -0x38(%rbp),%rax
0x00000000006cca16 <+559>: mov %edx,0x10(%rax)
0x00000000006cca19 <+562>: mov -0x38(%rbp),%rax
0x00000000006cca1d <+566>: mov 0x18(%rax),%rdx
0x00000000006cca21 <+570>: mov -0x40(%rbp),%rax
0x00000000006cca25 <+574>: mov 0x18(%rax),%rax
0x00000000006cca29 <+578>: add %rax,%rdx
0x00000000006cca2c <+581>: mov -0x38(%rbp),%rax
0x00000000006cca30 <+585>: mov %rdx,0x18(%rax)
0x00000000006cca34 <+589>: mov -0x40(%rbp),%rax
0x00000000006cca38 <+593>: mov %rax,%rdi
0x00000000006cca3b <+596>: callq 0x6c9edb <rd_kafka_msgq_init>
0x00000000006cca40 <+601>: leaveq
0x00000000006cca41 <+602>: retq
End of assembler dump.
(gdb)
(gdb) p *at
$13 = {rkm_rkmessage = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f95fc000c70, partition = 133, payload = 0x7f950402f5a0, len = 2287, key = 0x7f950402fe8f,
key_len = 29, offset = -1001, _private = 0x0}, rkm_link = {tqe_next = 0x7f950402ffa0, tqe_prev = 0x7f950402e878}, rkm_flags = 458754,
rkm_timestamp = 1534694858523, rkm_tstype = RD_KAFKA_TIMESTAMP_CREATE_TIME, rkm_headers = 0x7f950402f440, rkm_u = {producer = {ts_timeout = 8963336588783,
ts_enq = 8963036588783, ts_backoff = 8963038414997, msgseq = 411689, retries = 0}, consumer = {binhdrs = {len = -260157969, data = 0x826de9caaef,
_data = "\225"}}}}
(gdb)
$14 = {rkm_rkmessage = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f95fc000c70, partition = 133, payload = 0x7f950402f5a0, len = 2287, key = 0x7f950402fe8f,
key_len = 29, offset = -1001, _private = 0x0}, rkm_link = {tqe_next = 0x7f950402ffa0, tqe_prev = 0x7f950402e878}, rkm_flags = 458754,
rkm_timestamp = 1534694858523, rkm_tstype = RD_KAFKA_TIMESTAMP_CREATE_TIME, rkm_headers = 0x7f950402f440, rkm_u = {producer = {ts_timeout = 8963336588783,
ts_enq = 8963036588783, ts_backoff = 8963038414997, msgseq = 411689, retries = 0}, consumer = {binhdrs = {len = -260157969, data = 0x826de9caaef,
_data = "\225"}}}}
(gdb) p *destq
$15 = {rkmq_msgs = {tqh_first = 0x7f9504026290, tqh_last = 0x7f954c028f68}, rkmq_msg_cnt = 656, rkmq_msg_bytes = 865908}
(gdb) p destq.rkmq_msgs
$16 = {tqh_first = 0x7f9504026290, tqh_last = 0x7f954c028f68}
(gdb) p *first
$17 = {rkm_rkmessage = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f95fc000c70, partition = 133, payload = 0x7f94ec010710, len = 1240, key = 0x7f94ec010be8,
key_len = 28, offset = -1001, _private = 0x0}, rkm_link = {tqe_next = 0x7f94ec010cf0, tqe_prev = 0x7f950402f548}, rkm_flags = 458754,
rkm_timestamp = 1534694860034, rkm_tstype = RD_KAFKA_TIMESTAMP_CREATE_TIME, rkm_headers = 0x7f94ec010590, rkm_u = {producer = {ts_timeout = 8963338100537,
ts_enq = 8963038100537, ts_backoff = 8963038600256, msgseq = 412354, retries = 0}, consumer = {binhdrs = {len = -258646215, data = 0x826deb3bc39,
_data = "@"}}}}
(gdb) p *dest_first
$18 = {rkm_rkmessage = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f95fc000c70, partition = 133, payload = 0x7f950402f5a0, len = 2287, key = 0x7f950402fe8f,
key_len = 29, offset = -1001, _private = 0x0}, rkm_link = {tqe_next = 0x7f950402ffa0, tqe_prev = 0x7f950402e878}, rkm_flags = 458754,
rkm_timestamp = 1534694858523, rkm_tstype = RD_KAFKA_TIMESTAMP_CREATE_TIME, rkm_headers = 0x7f950402f440, rkm_u = {producer = {ts_timeout = 8963336588783,
ts_enq = 8963036588783, ts_backoff = 8963038414997, msgseq = 411689, retries = 0}, consumer = {binhdrs = {len = -260157969, data = 0x826de9caaef,
_data = "\225"}}}}
(gdb)
Fantastic! Can you provide info reg from that same frame too?
edenhill
on 23 Aug 2018
@edenhill
#0 0x00000000006cc9e7 in rd_kafka_msgq_insert_msgq (destq=0x7f95a424ed18,
srcq=0x7f9584fc4170, cmp=0x68bb70 <rd_kafka_msg_cmp_msgseq>)
at rdkafka_partition.c:733
733 in rdkafka_partition.c
(gdb) info reg
rax 0x0 0
rbx 0x0 0
rcx 0x7f950402f500 140277994157312
rdx 0x7f951804fb78 140278329834360
rsi 0x7f950402f500 140277994157312
rdi 0x7f94ec010670 140277591377520
rbp 0x7f9584fc4120 0x7f9584fc4120
rsp 0x7f9584fc40d0 0x7f9584fc40d0
r8 0x826debb5c40 8963038600256
r9 0x68bb70 6863728
r10 0x7f95fc000cbc 140282154716348
r11 0x1dd14e48 500256328
r12 0x0 0
r13 0x0 0
r14 0x7f9584fc99c0 140280158001600
r15 0x7f9584fc9700 140280158000896
rip 0x6cc9e7 0x6cc9e7 <rd_kafka_msgq_insert_msgq+512>
eflags 0x10202 [ IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
---Type <return> to continue, or q <return> to quit---
gs 0x0 0
(gdb)
❤1
In that core file again, can you move to frame 3:
#3 0x00000000006b56ab in rd_kafka_handle_Produce (rk=0x7f95e001e440, rkb=0x7f95a4100490, err=RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION, reply=0x7f93e80581f0, request=0x7f93e80583e0, opaque=0x7f95a424ec20) at rdkafka_request.c:1883
and print:
p *rktp
p *rktp->rktp_leader
p rkb
p *rkb
p *reply
p *reply->rkbuf_rkb
p *request
p *request->rkbuf_rkb
I'm suspecting the leader for the partition was changed (NOT_LEADER_FOR_PARTITION would indicate so as well :) ), so the response handler here is accessing the unprotected thread-local rktp->rktp_xmit_msgq queue which that thread no longer owns. Concurrent unprotected access :boom:
edenhill
on 23 Aug 2018
The retrythreadfix branch of librdkafka has a fix for this issue (if your gdb printout proves the theory right)
edenhill
on 23 Aug 2018
@edenhill
(gdb) frame 3
#3 0x00000000006b56ab in rd_kafka_handle_Produce (rk=0x7f95e001e440,
rkb=0x7f95a4100490, err=RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION,
reply=0x7f93e80581f0, request=0x7f93e80583e0, opaque=0x7f95a424ec20)
at rdkafka_request.c:1883
1883 rdkafka_request.c: No such file or directory.
(gdb) p *rktp
$1 = {rktp_rklink = {tqe_next = 0x0, tqe_prev = 0x0}, rktp_rkblink = {
tqe_next = 0x0, tqe_prev = 0x7f95a425b5f0}, rktp_activelink = {
cqe_next = 0x7f95a409f660, cqe_prev = 0x7f95a425b5e0}, rktp_rktlink = {
tqe_next = 0x0, tqe_prev = 0x0}, rktp_cgrplink = {tqe_next = 0x0,
tqe_prev = 0x0}, rktp_rkt = 0x7f95fc000c70, rktp_s_rkt = 0x7f95fc000c70,
rktp_partition = 133, rktp_leader_id = 75380, rktp_leader = 0x7f95a409f5d0,
rktp_next_leader = 0x0, rktp_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 3}, rktp_lock = {
__data = {__lock = 1, __count = 0, __owner = 13696, __nusers = 1,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}},
__size = "\001\000\000\000\000\000\000\000\200\065\000\000\001", '\000' <repeats 26 times>, __align = 1}, rktp_msgq_wakeup_fd = 118, rktp_msgq = {
rkmq_msgs = {tqh_first = 0x0, tqh_last = 0x7f95a424ecf8},
rkmq_msg_cnt = 0, rkmq_msg_bytes = 0}, rktp_xmit_msgq = {rkmq_msgs = {
tqh_first = 0x7f9504026290, tqh_last = 0x7f954c028f68},
rkmq_msg_cnt = 656, rkmq_msg_bytes = 865908}, rktp_fetch = 0,
rktp_fetchq = 0x7f95a424ef80, rktp_ops = 0x7f95a424f070,
rktp_msgseq = 412524, rktp_version = {val = 1}, rktp_op_version = 1,
rktp_fetch_version = 0, rktp_fetch_state = RD_KAFKA_TOPPAR_FETCH_NONE,
rktp_fetch_msg_max_bytes = 1048576, rktp_ts_fetch_backoff = 0,
---Type <return> to continue, or q <return> to quit---
rktp_query_offset = 0, rktp_next_offset = 0, rktp_last_next_offset = 0,
rktp_app_offset = -1001, rktp_stored_offset = -1001,
rktp_committing_offset = 0, rktp_committed_offset = -1001,
rktp_ts_committed_offset = 0, rktp_offsets = {fetch_offset = 0,
eof_offset = -1001, hi_offset = -1001}, rktp_offsets_fin = {
fetch_offset = 0, eof_offset = -1001, hi_offset = -1001},
rktp_hi_offset = -1001, rktp_lo_offset = -1001, rktp_ts_offset_lag = 0,
rktp_offset_path = 0x0, rktp_offset_fp = 0x0, rktp_cgrp = 0x0,
rktp_assigned = 0, rktp_replyq = {q = 0x0, version = 0, _id = 0x0},
rktp_flags = 0, rktp_s_for_desp = 0x0, rktp_s_for_cgrp = 0x0,
rktp_s_for_rkb = 0x7f95a424ec20, rktp_offset_query_tmr = {rtmr_link = {
tqe_next = 0x0, tqe_prev = 0x0}, rtmr_next = 0, rtmr_interval = 0,
rtmr_oneshot = 0 '\000', rtmr_callback = 0x0, rtmr_arg = 0x0},
rktp_offset_commit_tmr = {rtmr_link = {tqe_next = 0x0, tqe_prev = 0x0},
rtmr_next = 0, rtmr_interval = 0, rtmr_oneshot = 0 '\000',
rtmr_callback = 0x0, rtmr_arg = 0x0}, rktp_offset_sync_tmr = {rtmr_link = {
tqe_next = 0x0, tqe_prev = 0x0}, rtmr_next = 0, rtmr_interval = 0,
rtmr_oneshot = 0 '\000', rtmr_callback = 0x0, rtmr_arg = 0x0},
rktp_consumer_lag_tmr = {rtmr_link = {tqe_next = 0x0, tqe_prev = 0x0},
rtmr_next = 0, rtmr_interval = 0, rtmr_oneshot = 0 '\000',
rtmr_callback = 0x0, rtmr_arg = 0x0}, rktp_wait_consumer_lag_resp = 0,
rktp_c = {tx_msgs = {val = 413763}, tx_msg_bytes = {val = 545787250},
rx_msgs = {val = 0}, rx_msg_bytes = {val = 0}, producer_enq_msgs = {
---Type <return> to continue, or q <return> to quit---
val = 412524}, rx_ver_drops = {val = 0}}}
(gdb) p *rktp->rktp_leader
$2 = {rkb_link = {tqe_next = 0x7f95a40ad330, tqe_prev = 0x7f95a4091870},
rkb_nodeid = 75380, rkb_rsal = 0x7f949c1d4530,
rkb_ts_rsal_last = 8878017231260, rkb_addr_last = 0x7f949c1d4538,
rkb_transport = 0x0, rkb_corrid = 6073, rkb_connid = 3,
rkb_ops = 0x7f95a409fee0, rkb_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0},
rkb_blocking_max_ms = 13, rkb_toppars = {tqh_first = 0x7f95a422df20,
tqh_last = 0x7f95a424ec30}, rkb_toppar_cnt = 5, rkb_active_toppars = {
cqh_first = 0x7f95a422df20, cqh_last = 0x7f95a424ec20},
rkb_active_toppar_cnt = 5, rkb_active_toppar_next = 0x7f95a422df20,
rkb_cgrp = 0x0, rkb_ts_fetch_backoff = 0, rkb_fetching = 0,
rkb_state = RD_KAFKA_BROKER_STATE_UP, rkb_ts_state = 8878017600550,
rkb_timeout_scan_intvl = {ri_ts_last = 8963038098154, ri_fixed = 0,
ri_backoff = 0}, rkb_blocking_request_cnt = {val = 0},
rkb_features = 2047, rkb_ApiVersions = 0x7f949c1f2370,
rkb_ApiVersions_cnt = 34, rkb_ApiVersion_fail_intvl = {
ri_ts_last = 8878017500286, ri_fixed = 1200000000, ri_backoff = 0},
rkb_source = RD_KAFKA_LEARNED, rkb_c = {tx_bytes = {val = 422067884}, tx = {
val = 28659}, tx_err = {val = 0}, tx_retries = {val = 0},
req_timeouts = {val = 0}, rx_bytes = {val = 478178}, rx = {val = 6073},
rx_err = {val = 3}, rx_corrid_err = {val = 0}, rx_partial = {val = 0},
---Type <return> to continue, or q <return> to quit---
zbuf_grow = {val = 0}, buf_grow = {val = 0}, wakeups = {val = 1408259}},
rkb_req_timeouts = 0, rkb_ts_tx_last = 8879996402291,
rkb_ts_metadata_poll = 8719197400385, rkb_metadata_fast_poll_cnt = 0,
rkb_thread = 140280216749824, rkb_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 8},
rkb_rk = 0x7f95e001e440, rkb_recv_buf = 0x0, rkb_max_inflight = 1000000,
rkb_outbufs = {rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a409f7c8},
rkbq_cnt = {val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_waitresps = {
rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a409f7e0}, rkbq_cnt = {
val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_retrybufs = {rkbq_bufs = {
tqh_first = 0x0, tqh_last = 0x7f95a409f7f8}, rkbq_cnt = {val = 0},
rkbq_msg_cnt = {val = 0}}, rkb_avg_int_latency = {ra_v = {maxv = 1999356,
minv = 1684329, avg = 0, sum = 311815974, cnt = 166,
start = 8963019442607}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a40b15b0, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_outbuf_latency = {ra_v = {maxv = 0,
minv = 0, avg = 0, sum = 0, cnt = 0, start = 8963019442609}, ra_lock = {
---Type <return> to continue, or q <return> to quit---
__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0},
ra_enabled = 1, ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a409ffd0,
ra_hist = {p50 = 0, p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0,
oor = 0, hdrsize = 0, stddev = 0, mean = 0}}, rkb_avg_rtt = {ra_v = {
maxv = 0, minv = 0, avg = 0, sum = 0, cnt = 0, start = 8963019442611},
ra_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0},
ra_enabled = 1, ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a40a3450,
ra_hist = {p50 = 0, p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0,
oor = 0, hdrsize = 0, stddev = 0, mean = 0}}, rkb_avg_throttle = {
ra_v = {maxv = 0, minv = 0, avg = 0, sum = 0, cnt = 0,
start = 8963019442613}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a40a8d50, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}},
rkb_name = "ssl://kafka028.REDACTED.net:9093/75380", '\000' <repeats 82 times>,
---Type <return> to continue, or q <return> to quit---
rkb_nodename = "kafka028.REDACTED.net:9093", '\000' <repeats 94 times>, rkb_port = 9093,
rkb_origname = 0x7f95a409fe70 "kafka028.REDACTED.net",
rkb_logname = 0x7f95a409fea0 "ssl://kafka028.REDACTED.net:9093/75380", rkb_logname_lock = {__data = {__lock = 0, __count = 0, __owner = 0,
__nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>,
__align = 0}, rkb_wakeup_fd = {117, 118}, rkb_toppar_wakeup_fd = 118,
rkb_connect_intvl = {ri_ts_last = 8878017230322, ri_fixed = 0,
ri_backoff = 0}, rkb_proto = RD_KAFKA_PROTO_SSL, rkb_down_reported = 0,
rkb_err = {
msg = "ssl://kafka028.REDACTED.net:9093/75380: Disconnected", '\000' <repeats 452 times>, err = 0}}
(gdb) p rkb
$3 = (rd_kafka_broker_t *) 0x7f95a4100490
(gdb) p *rkb
$4 = {rkb_link = {tqe_next = 0x7f95a410e1f0, tqe_prev = 0x7f95a40f27a0},
rkb_nodeid = 63698, rkb_rsal = 0x7f93e8057670,
rkb_ts_rsal_last = 8963037011659, rkb_addr_last = 0x7f93e8057678,
rkb_transport = 0x7f93e80576a0, rkb_corrid = 12436, rkb_connid = 8,
rkb_ops = 0x7f95a4100da0, rkb_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0},
rkb_blocking_max_ms = 1000, rkb_toppars = {tqh_first = 0x7f95a420b130,
tqh_last = 0x7f95a4262970}, rkb_toppar_cnt = 2, rkb_active_toppars = {
cqh_first = 0x7f95a420b130, cqh_last = 0x7f95a4262960},
rkb_active_toppar_cnt = 2, rkb_active_toppar_next = 0x7f95a420b130,
rkb_cgrp = 0x0, rkb_ts_fetch_backoff = 0, rkb_fetching = 0,
rkb_state = RD_KAFKA_BROKER_STATE_UP, rkb_ts_state = 8963037363446,
rkb_timeout_scan_intvl = {ri_ts_last = 8963038000270, ri_fixed = 0,
ri_backoff = 0}, rkb_blocking_request_cnt = {val = 0},
rkb_features = 2047, rkb_ApiVersions = 0x7f93e80595b0,
rkb_ApiVersions_cnt = 34, rkb_ApiVersion_fail_intvl = {
ri_ts_last = 8963037300220, ri_fixed = 1200000000, ri_backoff = 0},
rkb_source = RD_KAFKA_LEARNED, rkb_c = {tx_bytes = {val = 649492492}, tx = {
val = 45781}, tx_err = {val = 0}, tx_retries = {val = 0},
req_timeouts = {val = 0}, rx_bytes = {val = 858113}, rx = {val = 12436},
rx_err = {val = 7}, rx_corrid_err = {val = 0}, rx_partial = {val = 0},
---Type <return> to continue, or q <return> to quit---
zbuf_grow = {val = 0}, buf_grow = {val = 0}, wakeups = {val = 2209509}},
rkb_req_timeouts = 0, rkb_ts_tx_last = 8963038316094,
rkb_ts_metadata_poll = 8719198025123, rkb_metadata_fast_poll_cnt = 0,
rkb_thread = 140280158000896, rkb_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 6},
rkb_rk = 0x7f95e001e440, rkb_recv_buf = 0x0, rkb_max_inflight = 1000000,
rkb_outbufs = {rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a4100688},
rkbq_cnt = {val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_waitresps = {
rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a41006a0}, rkbq_cnt = {
val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_retrybufs = {rkbq_bufs = {
tqh_first = 0x0, tqh_last = 0x7f95a41006b8}, rkbq_cnt = {val = 0},
rkbq_msg_cnt = {val = 0}}, rkb_avg_int_latency = {ra_v = {maxv = 1722480,
minv = 26, avg = 0, sum = 1659366616, cnt = 2146,
start = 8963019442705}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a40273f0, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_outbuf_latency = {ra_v = {maxv = 354499,
minv = 52, avg = 0, sum = 612280, cnt = 11, start = 8963019442707},
---Type <return> to continue, or q <return> to quit---
ra_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0},
ra_enabled = 1, ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a42f14a0,
ra_hist = {p50 = 0, p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0,
oor = 0, hdrsize = 0, stddev = 0, mean = 0}}, rkb_avg_rtt = {ra_v = {
maxv = 192096, minv = 63105, avg = 0, sum = 1219579, cnt = 11,
start = 8963019442709}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a4100e90, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_throttle = {ra_v = {maxv = 0, minv = 0,
avg = 0, sum = 0, cnt = 10, start = 8963019442710}, ra_lock = {__data = {
__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
__spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a4109c10, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}},
rkb_name = "ssl://kafka005.REDACTED.net:9093/63698", '\000' <repeats 82 times>,
---Type <return> to continue, or q <return> to quit---
rkb_nodename = "kafka005.REDACTED.net:9093", '\000' <repeats 94 times>, rkb_port = 9093,
rkb_origname = 0x7f95a4100d30 "kafka005.REDACTED.net",
rkb_logname = 0x7f95a4100d60 "ssl://kafka005.REDACTED.net:9093/63698", rkb_logname_lock = {__data = {__lock = 0, __count = 0, __owner = 0,
__nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>,
__align = 0}, rkb_wakeup_fd = {131, 132}, rkb_toppar_wakeup_fd = 132,
rkb_connect_intvl = {ri_ts_last = 8963037010020, ri_fixed = 0,
ri_backoff = 0}, rkb_proto = RD_KAFKA_PROTO_SSL, rkb_down_reported = 0,
rkb_err = {
msg = "ssl://kafka005.REDACTED.net:9093/63698: Disconnected", '\000' <repeats 452 times>, err = 0}}
(gdb) p *reply
$5 = {rkbuf_link = {tqe_next = 0x0, tqe_prev = 0x0}, rkbuf_corrid = 0,
rkbuf_ts_retry = 0, rkbuf_flags = 0, rkbuf_buf = {rbuf_segments = {
tqh_first = 0x7f93e8059750, tqh_last = 0x7f93e8059798},
rbuf_segment_cnt = 2, rbuf_wpos = 0x7f93e8059798, rbuf_len = 50,
rbuf_size = 50, rbuf_extra = 0x7f93e8059750 "\230\227\005\350\223\177",
rbuf_extra_len = 136, rbuf_extra_size = 136}, rkbuf_reader = {
buf = 0x7f93e8058218, seg = 0x7f93e8059798, rof = 42, start = 8,
end = 50}, rkbuf_connid = 0, rkbuf_totlen = 42, rkbuf_crc = 0,
rkbuf_reqhdr = {Size = 0, ApiKey = 0, ApiVersion = 3, CorrId = 0},
rkbuf_reshdr = {Size = 46, CorrId = 12436}, rkbuf_expected_size = 0,
rkbuf_replyq = {q = 0x0, version = 0, _id = 0x0}, rkbuf_orig_replyq = {
q = 0x0, version = 0, _id = 0x0}, rkbuf_cb = 0x0, rkbuf_response = 0x0,
rkbuf_rkb = 0x7f95a4100490, rkbuf_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 1},
rkbuf_opaque = 0x0, rkbuf_retries = 0, rkbuf_features = 0, rkbuf_ts_enq = 0,
rkbuf_ts_sent = 0, rkbuf_ts_timeout = 0, rkbuf_abs_timeout = 0,
rkbuf_rel_timeout = 0, rkbuf_offset = 0, rkbuf_rktp_vers = 0x0,
rkbuf_msgq = {rkmq_msgs = {tqh_first = 0x0, tqh_last = 0x7f93e8058378},
rkmq_msg_cnt = 0, rkmq_msg_bytes = 0},
rkbuf_err = RD_KAFKA_RESP_ERR_NO_ERROR, rkbuf_u = {Metadata = {topics = 0x0,
reason = 0x0, rko = 0x0, all_topics = 0, decr = 0x0, decr_lock = 0x0}},
---Type <return> to continue, or q <return> to quit---
rkbuf_uflow_mitigation = 0x0}
(gdb) p *reply->rkbuf_rkb
$6 = {rkb_link = {tqe_next = 0x7f95a410e1f0, tqe_prev = 0x7f95a40f27a0},
rkb_nodeid = 63698, rkb_rsal = 0x7f93e8057670,
rkb_ts_rsal_last = 8963037011659, rkb_addr_last = 0x7f93e8057678,
rkb_transport = 0x7f93e80576a0, rkb_corrid = 12436, rkb_connid = 8,
rkb_ops = 0x7f95a4100da0, rkb_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0},
rkb_blocking_max_ms = 1000, rkb_toppars = {tqh_first = 0x7f95a420b130,
tqh_last = 0x7f95a4262970}, rkb_toppar_cnt = 2, rkb_active_toppars = {
cqh_first = 0x7f95a420b130, cqh_last = 0x7f95a4262960},
rkb_active_toppar_cnt = 2, rkb_active_toppar_next = 0x7f95a420b130,
rkb_cgrp = 0x0, rkb_ts_fetch_backoff = 0, rkb_fetching = 0,
rkb_state = RD_KAFKA_BROKER_STATE_UP, rkb_ts_state = 8963037363446,
rkb_timeout_scan_intvl = {ri_ts_last = 8963038000270, ri_fixed = 0,
ri_backoff = 0}, rkb_blocking_request_cnt = {val = 0},
rkb_features = 2047, rkb_ApiVersions = 0x7f93e80595b0,
rkb_ApiVersions_cnt = 34, rkb_ApiVersion_fail_intvl = {
ri_ts_last = 8963037300220, ri_fixed = 1200000000, ri_backoff = 0},
rkb_source = RD_KAFKA_LEARNED, rkb_c = {tx_bytes = {val = 649492492}, tx = {
val = 45781}, tx_err = {val = 0}, tx_retries = {val = 0},
req_timeouts = {val = 0}, rx_bytes = {val = 858113}, rx = {val = 12436},
rx_err = {val = 7}, rx_corrid_err = {val = 0}, rx_partial = {val = 0},
---Type <return> to continue, or q <return> to quit---
zbuf_grow = {val = 0}, buf_grow = {val = 0}, wakeups = {val = 2209509}},
rkb_req_timeouts = 0, rkb_ts_tx_last = 8963038316094,
rkb_ts_metadata_poll = 8719198025123, rkb_metadata_fast_poll_cnt = 0,
rkb_thread = 140280158000896, rkb_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 6},
rkb_rk = 0x7f95e001e440, rkb_recv_buf = 0x0, rkb_max_inflight = 1000000,
rkb_outbufs = {rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a4100688},
rkbq_cnt = {val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_waitresps = {
rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a41006a0}, rkbq_cnt = {
val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_retrybufs = {rkbq_bufs = {
tqh_first = 0x0, tqh_last = 0x7f95a41006b8}, rkbq_cnt = {val = 0},
rkbq_msg_cnt = {val = 0}}, rkb_avg_int_latency = {ra_v = {maxv = 1722480,
minv = 26, avg = 0, sum = 1659366616, cnt = 2146,
start = 8963019442705}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a40273f0, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_outbuf_latency = {ra_v = {maxv = 354499,
minv = 52, avg = 0, sum = 612280, cnt = 11, start = 8963019442707},
---Type <return> to continue, or q <return> to quit---
ra_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0},
ra_enabled = 1, ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a42f14a0,
ra_hist = {p50 = 0, p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0,
oor = 0, hdrsize = 0, stddev = 0, mean = 0}}, rkb_avg_rtt = {ra_v = {
maxv = 192096, minv = 63105, avg = 0, sum = 1219579, cnt = 11,
start = 8963019442709}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a4100e90, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_throttle = {ra_v = {maxv = 0, minv = 0,
avg = 0, sum = 0, cnt = 10, start = 8963019442710}, ra_lock = {__data = {
__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
__spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a4109c10, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}},
rkb_name = "ssl://kafka005.REDACTED.net:9093/63698", '\000' <repeats 82 times>,
---Type <return> to continue, or q <return> to quit---
rkb_nodename = "kafka005.REDACTED.net:9093", '\000' <repeats 94 times>, rkb_port = 9093,
rkb_origname = 0x7f95a4100d30 "kafka005.REDACTED.net",
rkb_logname = 0x7f95a4100d60 "ssl://kafka005.REDACTED.net:9093/63698", rkb_logname_lock = {__data = {__lock = 0, __count = 0, __owner = 0,
__nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>,
__align = 0}, rkb_wakeup_fd = {131, 132}, rkb_toppar_wakeup_fd = 132,
rkb_connect_intvl = {ri_ts_last = 8963037010020, ri_fixed = 0,
ri_backoff = 0}, rkb_proto = RD_KAFKA_PROTO_SSL, rkb_down_reported = 0,
rkb_err = {
msg = "ssl://kafka005.REDACTED.net:9093/63698: Disconnected", '\000' <repeats 452 times>, err = 0}}
(gdb) p *request
$7 = {rkbuf_link = {tqe_next = 0x0, tqe_prev = 0x7f95a41006a0},
rkbuf_corrid = 12436, rkbuf_ts_retry = 0, rkbuf_flags = 0, rkbuf_buf = {
rbuf_segments = {tqh_first = 0x7f93e802cf40, tqh_last = 0x7f93e8045e58},
rbuf_segment_cnt = 3, rbuf_wpos = 0x7f93e8045e58, rbuf_len = 24867,
rbuf_size = 24903, rbuf_extra = 0x7f93e802cf40 "\230^\004\350\223\177",
rbuf_extra_len = 102296, rbuf_extra_size = 128375}, rkbuf_reader = {
buf = 0x7f93e8058408, seg = 0x7f93e8045e98, rof = 24753, start = 0,
end = 24867}, rkbuf_connid = 8, rkbuf_totlen = 0, rkbuf_crc = 0,
rkbuf_reqhdr = {Size = 0, ApiKey = 0, ApiVersion = 3, CorrId = 0},
rkbuf_reshdr = {Size = 0, CorrId = 0}, rkbuf_expected_size = 0,
rkbuf_replyq = {q = 0x0, version = 0, _id = 0x0}, rkbuf_orig_replyq = {
q = 0x0, version = 0, _id = 0x0},
rkbuf_cb = 0x6b5255 <rd_kafka_handle_Produce>, rkbuf_response = 0x0,
rkbuf_rkb = 0x7f95a4100490, rkbuf_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 1},
rkbuf_opaque = 0x7f95a424ec20, rkbuf_retries = 0, rkbuf_features = 512,
rkbuf_ts_enq = 8963038314357, rkbuf_ts_sent = 184101,
rkbuf_ts_timeout = 8963098314357, rkbuf_abs_timeout = 8963338100356,
rkbuf_rel_timeout = 0, rkbuf_offset = 0, rkbuf_rktp_vers = 0x0,
rkbuf_msgq = {rkmq_msgs = {tqh_first = 0x0, tqh_last = 0x7f93e8058568},
rkmq_msg_cnt = 0, rkmq_msg_bytes = 0},
---Type <return> to continue, or q <return> to quit---
rkbuf_err = RD_KAFKA_RESP_ERR_NO_ERROR, rkbuf_u = {Metadata = {topics = 0x0,
reason = 0x0, rko = 0x0, all_topics = 0, decr = 0x0, decr_lock = 0x0}},
rkbuf_uflow_mitigation = 0x0}
(gdb) p *request->rkbuf_rkb
$8 = {rkb_link = {tqe_next = 0x7f95a410e1f0, tqe_prev = 0x7f95a40f27a0},
rkb_nodeid = 63698, rkb_rsal = 0x7f93e8057670,
rkb_ts_rsal_last = 8963037011659, rkb_addr_last = 0x7f93e8057678,
rkb_transport = 0x7f93e80576a0, rkb_corrid = 12436, rkb_connid = 8,
rkb_ops = 0x7f95a4100da0, rkb_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0},
rkb_blocking_max_ms = 1000, rkb_toppars = {tqh_first = 0x7f95a420b130,
tqh_last = 0x7f95a4262970}, rkb_toppar_cnt = 2, rkb_active_toppars = {
cqh_first = 0x7f95a420b130, cqh_last = 0x7f95a4262960},
rkb_active_toppar_cnt = 2, rkb_active_toppar_next = 0x7f95a420b130,
rkb_cgrp = 0x0, rkb_ts_fetch_backoff = 0, rkb_fetching = 0,
rkb_state = RD_KAFKA_BROKER_STATE_UP, rkb_ts_state = 8963037363446,
rkb_timeout_scan_intvl = {ri_ts_last = 8963038000270, ri_fixed = 0,
ri_backoff = 0}, rkb_blocking_request_cnt = {val = 0},
rkb_features = 2047, rkb_ApiVersions = 0x7f93e80595b0,
rkb_ApiVersions_cnt = 34, rkb_ApiVersion_fail_intvl = {
ri_ts_last = 8963037300220, ri_fixed = 1200000000, ri_backoff = 0},
rkb_source = RD_KAFKA_LEARNED, rkb_c = {tx_bytes = {val = 649492492}, tx = {
val = 45781}, tx_err = {val = 0}, tx_retries = {val = 0},
req_timeouts = {val = 0}, rx_bytes = {val = 858113}, rx = {val = 12436},
rx_err = {val = 7}, rx_corrid_err = {val = 0}, rx_partial = {val = 0},
---Type <return> to continue, or q <return> to quit---
zbuf_grow = {val = 0}, buf_grow = {val = 0}, wakeups = {val = 2209509}},
rkb_req_timeouts = 0, rkb_ts_tx_last = 8963038316094,
rkb_ts_metadata_poll = 8719198025123, rkb_metadata_fast_poll_cnt = 0,
rkb_thread = 140280158000896, rkb_refcnt = {lock = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
__elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, v = 6},
rkb_rk = 0x7f95e001e440, rkb_recv_buf = 0x0, rkb_max_inflight = 1000000,
rkb_outbufs = {rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a4100688},
rkbq_cnt = {val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_waitresps = {
rkbq_bufs = {tqh_first = 0x0, tqh_last = 0x7f95a41006a0}, rkbq_cnt = {
val = 0}, rkbq_msg_cnt = {val = 0}}, rkb_retrybufs = {rkbq_bufs = {
tqh_first = 0x0, tqh_last = 0x7f95a41006b8}, rkbq_cnt = {val = 0},
rkbq_msg_cnt = {val = 0}}, rkb_avg_int_latency = {ra_v = {maxv = 1722480,
minv = 26, avg = 0, sum = 1659366616, cnt = 2146,
start = 8963019442705}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a40273f0, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_outbuf_latency = {ra_v = {maxv = 354499,
minv = 52, avg = 0, sum = 612280, cnt = 11, start = 8963019442707},
---Type <return> to continue, or q <return> to quit---
ra_lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0},
ra_enabled = 1, ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a42f14a0,
ra_hist = {p50 = 0, p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0,
oor = 0, hdrsize = 0, stddev = 0, mean = 0}}, rkb_avg_rtt = {ra_v = {
maxv = 192096, minv = 63105, avg = 0, sum = 1219579, cnt = 11,
start = 8963019442709}, ra_lock = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a4100e90, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}}, rkb_avg_throttle = {ra_v = {maxv = 0, minv = 0,
avg = 0, sum = 0, cnt = 10, start = 8963019442710}, ra_lock = {__data = {
__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
__spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, ra_enabled = 1,
ra_type = RD_AVG_GAUGE, ra_hdr = 0x7f95a4109c10, ra_hist = {p50 = 0,
p75 = 0, p90 = 0, p95 = 0, p99 = 0, p99_99 = 0, oor = 0, hdrsize = 0,
stddev = 0, mean = 0}},
rkb_name = "ssl://kafka005.REDACTED.net:9093/63698", '\000' <repeats 82 times>,
---Type <return> to continue, or q <return> to quit---
rkb_nodename = "kafka005.REDACTED.net:9093", '\000' <repeats 94 times>, rkb_port = 9093,
rkb_origname = 0x7f95a4100d30 "kafka005.REDACTED.net",
rkb_logname = 0x7f95a4100d60 "ssl://kafka005.REDACTED.net:9093/63698", rkb_logname_lock = {__data = {__lock = 0, __count = 0, __owner = 0,
__nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>,
__align = 0}, rkb_wakeup_fd = {131, 132}, rkb_toppar_wakeup_fd = 132,
rkb_connect_intvl = {ri_ts_last = 8963037010020, ri_fixed = 0,
ri_backoff = 0}, rkb_proto = RD_KAFKA_PROTO_SSL, rkb_down_reported = 0,
rkb_err = {
msg = "ssl://kafka005.REDACTED.net:9093/63698: Disconnected", '\000' <repeats 452 times>, err = 0}}
(gdb)
Partition leader is 75380, while the request was sent to, and response received from, broker 63698, so that means the theory stands and the fix is valid!
Would you be able to try out the fix as soon as it lands in master?
edenhill
on 23 Aug 2018
Absolutely!
On Thu, Aug 23, 2018, 8:58 AM Magnus Edenhill notifications@github.com
wrote:
Partition leader is 75380, while the request was sent to, and response
received from, broker 63698, so that means the theory stands and the fix is
valid!Would you be able to try out the fix as soon as it lands in master?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/confluent-kafka-go/issues/203#issuecomment-415470249,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABoBElP19BECLyUuvg_rPYsYhLOc1RBjks5uTtE0gaJpZM4Um7tI
.
billygout
on 23 Aug 2018
@billygout The fix is now on librdkafka master, fire away!
edenhill
on 23 Aug 2018
ok, it'll take us about a week to get it out, and another week of no crashes (hopefully) to claim victory.
billygout
on 23 Aug 2018
Do you want me to craft an RC to make it a bit more formal?
edenhill
on 23 Aug 2018
I personally don't mind, so it's up to you.
billygout
on 23 Aug 2018
Let me create an RC after some release testing tomorrow, and you can use that.
Thanks
edenhill
on 23 Aug 2018
In any case, i've been rebasing https://github.com/gduranceau/librdkafka/tree/connect_on_demand on top of master.
billygout
on 23 Aug 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
vanipolnedi
·
10Comments
mnishizawa
·
5Comments
binary132
·
10Comments
QuanVan95
·
4Comments
artemyarulin
·
8Comments