Fluent-bit: Memory Leak in DataDog output plugin

[PettitWesley]

First reported here: https://github.com/aws/aws-for-fluent-bit/issues/56

Valgrind confirms that there is a memory leak:

HEAP SUMMARY:
==7319==     in use at exit: 3,407 bytes in 7 blocks
==7319==   total heap usage: 20,783 allocs, 20,777 frees, 7,099,871 bytes allocated
==7319==
==7319== 47 bytes in 1 blocks are definitely lost in loss record 1 of 2
==7319==    at 0x4C2BB7B: malloc (vg_replace_malloc.c:299)
==7319==    by 0x43EDEC: flb_malloc (flb_mem.h:62)
==7319==    by 0x43F00B: sds_alloc (flb_sds.c:40)
==7319==    by 0x43F08B: flb_sds_create_len (flb_sds.c:61)
==7319==    by 0x43F12D: flb_sds_create (flb_sds.c:87)
==7319==    by 0x4A5A8D: flb_datadog_conf_create (datadog_conf.c:152)
==7319==    by 0x4A41AF: cb_datadog_init (datadog.c:41)
==7319==    by 0x445059: flb_output_init_all (flb_output.c:714)
==7319==    by 0x44EE23: flb_engine_start (flb_engine.c:466)
==7319==    by 0x433284: flb_main (fluent-bit.c:1035)
==7319==    by 0x4332D2: main (fluent-bit.c:1048)
==7319==
==7319== 3,360 bytes in 6 blocks are definitely lost in loss record 2 of 2
==7319==    at 0x4C2BB7B: malloc (vg_replace_malloc.c:299)
==7319==    by 0x43EDEC: flb_malloc (flb_mem.h:62)
==7319==    by 0x43F00B: sds_alloc (flb_sds.c:40)
==7319==    by 0x43F147: flb_sds_create_size (flb_sds.c:92)
==7319==    by 0x4A45B1: datadog_format (datadog.c:158)
==7319==    by 0x4A4C88: cb_datadog_flush (datadog.c:299)
==7319==    by 0x44F98C: output_pre_cb_flush (flb_output.h:449)
==7319==    by 0x708A66: co_init (amd64.c:117)
==7319==
==7319== LEAK SUMMARY:
==7319==    definitely lost: 3,407 bytes in 7 blocks
==7319==    indirectly lost: 0 bytes in 0 blocks
==7319==      possibly lost: 0 bytes in 0 blocks
==7319==    still reachable: 0 bytes in 0 blocks
==7319==         suppressed: 0 bytes in 0 blocks

[PettitWesley]

Alloc is happening here: https://github.com/fluent/fluent-bit/blob/master/plugins/out_datadog/datadog.c#L158

It seems to be allocating untracked memory every single flush, because the longer I run Fluent Bit, the more memory is lost.

Also, Valgrind flags a bad access on shut down:

==12212== Invalid free() / delete / delete[] / realloc()
==12212==    at 0x4C2CD28: free (vg_replace_malloc.c:530)
==12212==    by 0x4A5220: flb_free (flb_mem.h:122)
==12212==    by 0x4A5FCA: flb_datadog_conf_destroy (datadog_conf.c:228)
==12212==    by 0x4A5158: cb_datadog_exit (datadog.c:393)
==12212==    by 0x443DD7: flb_output_exit (flb_output.c:193)
==12212==    by 0x44F5D0: flb_engine_shutdown (flb_engine.c:655)
==12212==    by 0x43202B: flb_signal_handler (fluent-bit.c:391)
==12212==    by 0x5C18B4F: ??? (in /usr/lib64/libc-2.26.so)
==12212==    by 0x5CD1FDF: epoll_pwait (in /usr/lib64/libc-2.26.so)
==12212==    by 0x7064FA: _mk_event_wait (mk_event_epoll.c:354)
==12212==    by 0x7067EA: mk_event_wait (mk_event.c:163)
==12212==    by 0x44F0F6: flb_engine_start (flb_engine.c:555)
==12212==  Address 0x7cc70e0 is 16 bytes inside a block of size 47 alloc'd
==12212==    at 0x4C2BB7B: malloc (vg_replace_malloc.c:299)
==12212==    by 0x43EDEC: flb_malloc (flb_mem.h:62)
==12212==    by 0x43F00B: sds_alloc (flb_sds.c:40)
==12212==    by 0x43F08B: flb_sds_create_len (flb_sds.c:61)
==12212==    by 0x43F12D: flb_sds_create (flb_sds.c:87)
==12212==    by 0x4A5A8D: flb_datadog_conf_create (datadog_conf.c:152)
==12212==    by 0x4A41AF: cb_datadog_init (datadog.c:41)
==12212==    by 0x445059: flb_output_init_all (flb_output.c:714)
==12212==    by 0x44EE23: flb_engine_start (flb_engine.c:466)
==12212==    by 0x433284: flb_main (fluent-bit.c:1035)
==12212==    by 0x4332D2: main (fluent-bit.c:1048)

[edsiper]

Issue has been fixed.

[angulito]

Thank you so much @PettitWesley and @edsiper!! and thanks for your speed! do know when could we have a release that includes these changes (approx)? THANK YOU! 🥇

[edsiper]

We release v1.5.2 with the fixes tomorrow

On Thu, Jul 23, 2020, 06:23 AnguloHerrera [e-mail] wrote:

Thank you so much @PettitWesley https://github.com/PettitWesley and
@edsiper https://github.com/edsiper!! and thanks for your speed! do
know when could we have a release that includes these changes (approx)?
THANK YOU! 🥇

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluent-bit/issues/2379#issuecomment-662976116,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC2INTIKYHRC5YFPV272BTR5ATTRANCNFSM4PEICAUQ
.