Fluent-bit: FluentBit crash on Windows

Created on 12 Jun 2020 · 23Comments · Source: fluent/fluent-bit

Bug Report

Describe the bug

Following up from https://github.com/fluent/fluent-bit/commit/56d619fa73f2be9f0b7a5ae040d86ef0b9734a1c#commitcomment-39848013, I tried running the current latest fluent-bit, version 1.4.6, but it crashed a few times several minutes after starting (as a Windows service with winsw).

Your Environment

Version used: 1.4.6
Configuration: See below
Environment name and version: AWS EC2 instance
Server type and version: Windows
Operating System and version: Server 2016
Filters and plugins: in_tail, filter_lua, output_es

fluent-bit.conf:

[SERVICE]
    daemon                  false
    flush                   1
    log_level               debug
    parsers_file            parsers.conf
    plugins_file            plugins.conf
    http_server             true
    http_listen             0.0.0.0
    http_port               2020

[INPUT]
    name                    tail
    tag                     pharos.*
    path                    C:\PharosSystems\Logs\Json\PharosMpsLog.txt
    db                      C:\PharosSystems\Logs\Json\FluentBit.db
    parser                  json
    buffer_chunk_size       64k
    buffer_max_size         64k
    mem_buf_limit           8m
    skip_long_lines         true
    refresh_interval        10

[FILTER]
    name                    lua
    match                   *
    script                  filters.lua
    call                    transform

[OUTPUT]
    name                    es
    match                   *
    host                    logs
    port                    80
    logstash_format         true
    type                    _doc
    replace_dots            true
    retry_limit             5
    trace_error             true

filters.lua:

function transform(tag, timestamp, record)
    if record["Payload"] then
        record["Service"] = record["Payload"]["processName"]
    end

    if not record["Message"] and record["Payload"]["message"] then
        record["Message"], record["Payload"]["message"] = record["Payload"]["message"], nil
    end
    return 1, timestamp, record
end

Dump file captured with Sysinternals ProcDump using procdump64.exe -ma -e fluent-bit.exe:

fluent-bit.exe_200612_053456.zip

Note: missing symbols for fluent-bit.exe; @fujimotos could we please include PDB symbols with future Windows build artifacts? 🙏

Source

gitfool

All 23 comments

I'm looking at this issue. WFM.

fujimotos on 12 Jun 2020

@fujimotos I don't expect you to be able to repro the crash, but the captured dump should reveal the root cause, unless you don't have access to symbols either?

gitfool on 12 Jun 2020

but the captured dump should reveal the root cause, unless you don't have access to symbols either?

@gitfool No, we are building releases on AppVeyor, and haven't
been enabled PDB generation on these builds, whereas I admit
we really should have enabled it.

So I spent some time this morning and submit #2260. Every
Windows releases hereafter will have a corresponding PDB file.

Talking about the next step, I also can provide you a build that have
commit on top of v1.4.6. Can you check it out on your environment
if I provide one?

fujimotos on 15 Jun 2020

@fujimotos sure will, thanks!

gitfool on 15 Jun 2020

@fujimotos Meanwhile, I managed to build a debug build of 1.4.6 with VS Code and captured another crash dump:

fluent-bit.exe_200615_074931.zip (incl. fluent-bit.exe symbols)

I had a quick look around with WinDbg (from my Windows 10 machine):

Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [D:\Work\Support\FluentBit\fluent-bit.exe_200615_074931\fluent-bit.exe_200615_074931.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: '
*** "C:\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION'
Symbol search path is: srv*
Executable search path is: 
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)
Machine Name:
Debug session time: Mon Jun 15 19:49:31.000 2020 (UTC + 12:00)
System Uptime: 3 days 18:49:41.506
Process Uptime: 0 days 2:22:44.000
.................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(e38.12bc): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for fluent-bit.exe
fluent_bit!msgpack_unpacker_expand_buffer+0x624d6:
00007ff7`44c74966 8a06            mov     al,byte ptr [rsi] ds:000000c4`fbe00000=??
0:003> .sympath srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*d:\symbols*http://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
*** WARNING: Unable to verify checksum for fluent-bit.exe
0:003> .reload /f
.*** WARNING: Unable to verify checksum for fluent-bit.exe
................................

************* Symbol Loading Error Summary **************
Module name            Error
fluent-bit             No error - symbol load deferred
TmUmEvt64              The system cannot find the file specified
tmmon64                The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:003> .symopt+ 0x40
Symbol options are 0x30377:
  0x00000001 - SYMOPT_CASE_INSENSITIVE
  0x00000002 - SYMOPT_UNDNAME
  0x00000004 - SYMOPT_DEFERRED_LOADS
  0x00000010 - SYMOPT_LOAD_LINES
  0x00000020 - SYMOPT_OMAP_FIND_NEAREST
  0x00000040 - SYMOPT_LOAD_ANYTHING
  0x00000100 - SYMOPT_NO_UNQUALIFIED_LOADS
  0x00000200 - SYMOPT_FAIL_CRITICAL_ERRORS
  0x00010000 - SYMOPT_AUTO_PUBLICS
  0x00020000 - SYMOPT_NO_IMAGE_SEARCH
0:003> .reload /f
.*** WARNING: Unable to verify checksum for fluent-bit.exe
................................

************* Symbol Loading Error Summary **************
Module name            Error
TmUmEvt64              The system cannot find the file specified
tmmon64                The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:003> lm
start             end                 module name
00000000`6df10000 00000000`6e074000   tmmon64    (no symbols)           
00007ff7`44930000 00007ff7`44e44000   fluent_bit C (private pdb symbols)  D:\Devel\Repos\fluent-bit\build\bin\Debug\fluent-bit.pdb
00007ffd`c0490000 00007ffd`c04a6000   NapiNSP    (pdb symbols)          d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffd`c06d0000 00007ffd`c06de000   winrnr     (pdb symbols)          d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffd`c9b60000 00007ffd`c9cdc000   TmUmEvt64   (export symbols)       TmUmEvt64.dll
00007ffd`ca6d0000 00007ffd`ca6da000   rasadhlp   (pdb symbols)          d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffd`cc370000 00007ffd`cc3da000   FWPUCLNT   (pdb symbols)          d:\symbols\fwpuclnt.pdb\76287BDA6CE44E9F888125A7A291D1A81\fwpuclnt.pdb
00007ffd`ccf50000 00007ffd`ccf68000   nlaapi     (pdb symbols)          d:\symbols\nlaapi.pdb\2A36CC2DEED5441E824317E2B1D3B2DB1\nlaapi.pdb
00007ffd`ceb00000 00007ffd`ceb38000   IPHLPAPI   (pdb symbols)          d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffd`ceb40000 00007ffd`cebe2000   dnsapi     (pdb symbols)          d:\symbols\dnsapi.pdb\D92889441F4D4247AE4783B49D90B96F1\dnsapi.pdb
00007ffd`cedc0000 00007ffd`cee1c000   mswsock    (pdb symbols)          d:\symbols\mswsock.pdb\3ECFE529FA4746D98F4633C33EE460691\mswsock.pdb
00007ffd`cf440000 00007ffd`cf46b000   bcrypt     (pdb symbols)          d:\symbols\bcrypt.pdb\CF5F9E7894F74E0EB9157BD02BF123471\bcrypt.pdb
00007ffd`cf550000 00007ffd`cf560000   msasn1     (pdb symbols)          d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffd`cf580000 00007ffd`cf58f000   kernel_appcore   (pdb symbols)          d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffd`cf5f0000 00007ffd`cf771000   gdi32full   (pdb symbols)          d:\symbols\gdi32full.pdb\8C947FDC7D2B41BBACEBAE54C37B264F1\gdi32full.pdb
00007ffd`cf780000 00007ffd`cf874000   ucrtbase   (pdb symbols)          d:\symbols\ucrtbase.pdb\327103177C994904BA5C2BBBE08EC3331\ucrtbase.pdb
00007ffd`d01b0000 00007ffd`d01ce000   win32u     (pdb symbols)          d:\symbols\win32u.pdb\D746BAF2F0C04D5E9781C9CC9B12A2ED1\win32u.pdb
00007ffd`d01d0000 00007ffd`d03ed000   KERNELBASE   (pdb symbols)          d:\symbols\kernelbase.pdb\29BF43663E254495A4032C1C183AF4051\kernelbase.pdb
00007ffd`d03f0000 00007ffd`d05b9000   crypt32    (pdb symbols)          d:\symbols\crypt32.pdb\1CFEB418E4044D2FBAE01F87728DBEDA1\crypt32.pdb
00007ffd`d05c0000 00007ffd`d062a000   bcryptPrimitives   (pdb symbols)          d:\symbols\bcryptprimitives.pdb\A62F3617FC244277ADD68320C00250E91\bcryptprimitives.pdb
00007ffd`d0630000 00007ffd`d0796000   user32     (pdb symbols)          d:\symbols\user32.pdb\1E4B4E72316D49B2B245B9BA4F43265C1\user32.pdb
00007ffd`d07a0000 00007ffd`d084c000   kernel32   (pdb symbols)          d:\symbols\kernel32.pdb\FCB817ED2B6841C18F932CB4593BE1B71\kernel32.pdb
00007ffd`d0b80000 00007ffd`d0ca1000   rpcrt4     (pdb symbols)          d:\symbols\rpcrt4.pdb\526C08E00099481C96895AADD9B06CF01\rpcrt4.pdb
00007ffd`d0cb0000 00007ffd`d0d1a000   ws2_32     (pdb symbols)          d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffd`d0d20000 00007ffd`d0d28000   nsi        (pdb symbols)          d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffd`d0e90000 00007ffd`d0ec4000   gdi32      (pdb symbols)          d:\symbols\gdi32.pdb\3EE4BC8FDF37432694027B3713C3EF1C1\gdi32.pdb
00007ffd`d0f70000 00007ffd`d1235000   combase    (private pdb symbols)  d:\symbols\combase.pdb\F9E53A74EB3C4281A0A00B5F4B29E0011\combase.pdb
00007ffd`d12a0000 00007ffd`d12f2000   shlwapi    (pdb symbols)          d:\symbols\shlwapi.pdb\FB0E36D3E9774E3082EEF8F877CAE8721\shlwapi.pdb
00007ffd`d13f0000 00007ffd`d1492000   advapi32   (pdb symbols)          d:\symbols\advapi32.pdb\D8A05A9AD2FD4C3185A4041390FE54011\advapi32.pdb
00007ffd`d2de0000 00007ffd`d2e39000   sechost    (pdb symbols)          d:\symbols\sechost.pdb\3EE0CC142F5246A2A0879E93A04D88341\sechost.pdb
00007ffd`d2ff0000 00007ffd`d308e000   msvcrt     (pdb symbols)          d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffd`d3090000 00007ffd`d3098000   psapi      (pdb symbols)          d:\symbols\psapi.pdb\AEDA310D7062406A986FCE25C506506C1\psapi.pdb
00007ffd`d30a0000 00007ffd`d3270000   ntdll      (pdb symbols)          d:\symbols\ntdll.pdb\7B14B2F89F2D46F5998EFC7AF0F6EE0B1\ntdll.pdb
0:003> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : AV.Fault
    Value: Read

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 326981

    Key  : Timeline.Process.Start.DeltaSec
    Value: 8564


PROCESSES_ANALYSIS: 1

SERVICE_ANALYSIS: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1

Timeline: !analyze.Start
    Name: <blank>
    Time: 2020-06-15T08:53:28.507Z
    Diff: 3837507 mSec

Timeline: Dump.Current
    Name: <blank>
    Time: 2020-06-15T07:49:31.0Z
    Diff: 0 mSec

Timeline: Process.Start
    Name: <blank>
    Time: 2020-06-15T05:26:47.0Z
    Diff: 8564000 mSec

Timeline: OS.Boot
    Name: <blank>
    Time: 2020-06-11T12:59:50.0Z
    Diff: 326981000 mSec


DUMP_CLASS: 2

DUMP_QUALIFIER: 400

CONTEXT:  (.ecxr)
rax=000000c4fbdfe4cf rbx=000000c4fbdfd65a rcx=0000000000000000
rdx=0000000000000000 rsi=000000c4fbe00000 rdi=000000c4fbdfe530
rip=00007ff744c74966 rsp=000000c4fbdfd090 rbp=000000c5614454e1
 r8=000000c4fbdfd018  r9=000000c5614454e1 r10=0000000000000000
r11=0000000000000246 r12=0000000000000002 r13=0000000000000000
r14=000000000000036c r15=0000000000000012
iopl=0         nv up ei ng nz ac pe cy
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010291
fluent_bit!get_banner_text+0x2:
00007ff7`44c74966 8a06            mov     al,byte ptr [rsi] ds:000000c4`fbe00000=??
Resetting default scope

FAULTING_IP: 
fluent_bit!get_banner_text+2 [minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 91]
00007ff7`44c74966 8a06            mov     al,byte ptr [rsi]

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff744c74966 (fluent_bit!get_banner_text+0x0000000000000002)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 000000c4fbe00000
Attempt to read from address 000000c4fbe00000

DEFAULT_BUCKET_ID:  INVALID_POINTER_READ

PROCESS_NAME:  fluent-bit.exe

FOLLOWUP_IP: 
fluent_bit!get_banner_text+2 [minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 91]
00007ff7`44c74966 8a06            mov     al,byte ptr [rsi]

READ_ADDRESS:  000000c4fbe00000 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  000000c4fbe00000

WATSON_BKT_PROCSTAMP:  5ee70389

WATSON_BKT_MODULE:  fluent-bit.exe

WATSON_BKT_MODSTAMP:  5ee70389

WATSON_BKT_MODOFFSET:  344966

BUILD_VERSION_STRING:  10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)

MODLIST_WITH_TSCHKSUM_HASH:  cb609d3ddc950c61c43e6210cafcd09b72ad70c5

MODLIST_SHA1_HASH:  afc6ba214a6e2f01bd2716f01521ec8f38e36019

COMMENT:  
*** "C:\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

PRODUCT_TYPE:  3

SUITE_MASK:  400

DUMP_FLAGS:  8000c07

DUMP_TYPE:  3

ANALYSIS_SESSION_HOST:  HAL

ANALYSIS_SESSION_TIME:  06-15-2020 20:53:28.0507

ANALYSIS_VERSION: 10.0.18362.1 amd64fre

THREAD_ATTRIBUTES: 
OS_LOCALE:  ENU

BUGCHECK_STR:  APPLICATION_FAULT_INVALID_POINTER_READ

PRIMARY_PROBLEM_CLASS:  APPLICATION_FAULT

PROBLEM_CLASSES: 

    ID:     [0n313]
    Type:   [@ACCESS_VIOLATION]
    Class:  Addendum
    Scope:  BUCKET_ID
    Name:   Omit
    Data:   Omit
    PID:    [Unspecified]
    TID:    [0x12bc]
    Frame:  [0] : fluent_bit!get_banner_text

    ID:     [0n285]
    Type:   [INVALID_POINTER_READ]
    Class:  Primary
    Scope:  DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
            BUCKET_ID
    Name:   Add
    Data:   Omit
    PID:    [Unspecified]
    TID:    [0x12bc]
    Frame:  [0] : fluent_bit!get_banner_text

LAST_CONTROL_TRANSFER:  from 00007ff744c74f7d to 00007ff744c74966

STACK_TEXT:  
000000c4`fbdfd090 00007ff7`44c74f7d : 00000000`00000000 00000000`000001e4 000000c4`fbdfe560 00000000`00000000 : fluent_bit!get_banner_text+0x2
000000c4`fbdfe500 00007ff7`44c74d2f : 00000000`00001000 000000c4`fbdfe5e4 000000c4`fbdfe648 00000000`00000002 : fluent_bit!__acrt_convert_wcs_mbs_cp<char,wchar_t,<lambda_7c9dea7b4ca7285d2cdb541a38da6275>,__crt_win32_buffer_internal_dynamic_resizing>+0xb9
000000c4`fbdfe5a0 00007ff7`4495d486 : cccccccc`00000002 00000285`e1b87ed0 00000000`00000000 cccccccc`cccccccc : fluent_bit!__crt_seh_guarded_call<int>::operator()<<lambda_96b0dce3ad2a93c5c9ed548730f377b6>,<lambda_400f06e00cd32245fa85628833e911d2> &,<lambda_da1c5379cfd7439e775ba9665cd8bf87> >+0xbf
000000c4`fbdfe5f0 00007ff7`4495d5dd : 000000c4`fbdfe670 00000285`e1b85670 cccccccc`cccccccc cccccccc`00000000 : fluent_bit!gmtime_r+0x36
000000c4`fbdfe630 00007ff7`4495d76e : 00000000`000001d4 00000285`e1b85670 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!msgpack_pack_uint64+0x8d
000000c4`fbdff6a0 00007ff7`4495c387 : 00000285`e1b85670 00000285`e1b84240 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!msgpack_pack_uint64+0x21e
000000c4`fbdff700 00007ff7`44bed904 : 00000285`e1b84240 00000000`00000000 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!flb_hash_create+0x97
000000c4`fbdff740 00007ff7`44c58b90 : 00000285`e1b8f420 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!ecp_modp+0x134
000000c4`fbdff780 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!__crt_stdio_output::output_processor<char,__crt_stdio_output::stream_output_adapter<char>,__crt_stdio_output::format_validation_base<char,__crt_stdio_output::stream_output_adapter<char> > >::state_case_normal_tchar+0x64
000000c4`fbdff7b0 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
000000c4`fbdff7e0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


THREAD_SHA1_HASH_MOD_FUNC:  f60713582beccdfd0fb7be4255a1ec9df718ea5c

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  de28ed136a45017de4c000cd9538ba7bb9eacfcd

THREAD_SHA1_HASH_MOD:  86e11f2cb1bfae317fbe4a25f4030f00f1d95f28

FAULT_INSTR_CODE:  ff48068a

FAULTING_SOURCE_LINE:  minkernel\crts\ucrt\src\appcrt\startup\assert.cpp

FAULTING_SOURCE_FILE:  minkernel\crts\ucrt\src\appcrt\startup\assert.cpp

FAULTING_SOURCE_LINE_NUMBER:  91

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  fluent_bit!get_banner_text+2

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: fluent_bit

IMAGE_NAME:  fluent-bit.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  5ee70389

STACK_COMMAND:  dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~3s ; .ecxr ; kb

FAILURE_BUCKET_ID:  INVALID_POINTER_READ_c0000005_fluent-bit.exe!get_banner_text

BUCKET_ID:  APPLICATION_FAULT_INVALID_POINTER_READ_fluent_bit!get_banner_text+2

FAILURE_EXCEPTION_CODE:  c0000005

FAILURE_IMAGE_NAME:  fluent-bit.exe

BUCKET_ID_IMAGE_STR:  fluent-bit.exe

FAILURE_MODULE_NAME:  fluent_bit

BUCKET_ID_MODULE_STR:  fluent_bit

FAILURE_FUNCTION_NAME:  get_banner_text

BUCKET_ID_FUNCTION_STR:  get_banner_text

BUCKET_ID_OFFSET:  2

BUCKET_ID_MODTIMEDATESTAMP:  5ee70389

BUCKET_ID_MODCHECKSUM:  0

BUCKET_ID_MODVER_STR:  0.0.0.0

BUCKET_ID_PREFIX_STR:  APPLICATION_FAULT_INVALID_POINTER_READ_

FAILURE_PROBLEM_CLASS:  APPLICATION_FAULT

FAILURE_SYMBOL_NAME:  fluent-bit.exe!get_banner_text

TARGET_TIME:  2020-06-15T07:49:31.000Z

OSBUILD:  14393

OSSERVICEPACK:  2969

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 Server TerminalServer DataCenter SingleUserTS

USER_LCID:  0

OSBUILD_TIMESTAMP:  2019-05-04 16:40:41

BUILDDATESTAMP_STR:  180806-1810

BUILDLAB_STR:  rs1_release_inmarket_aim

BUILDOSVER_STR:  10.0.14393.2430

ANALYSIS_SESSION_ELAPSED_TIME:  45c

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:invalid_pointer_read_c0000005_fluent-bit.exe!get_banner_text

FAILURE_ID_HASH:  {bd2b5ce3-33ad-fe10-5574-3dab0cbd3dbc}

Followup:     MachineOwner
---------

0:003> ~*kb

   0  Id: e38.86c Suspend: 0 Teb: 000000c4`fb771000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`cedcc268 : 000000c4`fb8fe228 00000000`00000001 00000000`00000000 00000285`e1ae8de0 : ntdll!NtDeviceIoControlFile+0x14
01 00007ffd`d0cbc687 : 00000000`000001d8 00000285`e1b7e6f0 00000000`00000000 000000c4`fb8fe274 : mswsock!WSPSend+0x1c8
02 00007ff7`4495d07d : 00000285`e1b5fe30 00000000`00000000 00000000`00000000 000000c4`fb8ff460 : ws2_32!send+0x167
03 00007ff7`4497ac1b : cccccccc`00000004 00000000`00000000 cccccccc`00000000 00007ff7`44dc6ea8 : fluent_bit!flb_hash_entry_free+0x5d [D:\Devel\Repos\fluent-bit\src\flb_hash.c @ 98] 
04 00007ff7`4496bac5 : 00000285`e1b7c830 00000285`00000001 cccccccc`cccccccc cccccccc`00000000 : fluent_bit!flb_io_net_connect+0x20b [D:\Devel\Repos\fluent-bit\src\flb_io.c @ 127] 
05 00007ff7`4496bed9 : 00000000`00000200 00000285`e1b6f220 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!flb_config_init+0x85 [D:\Devel\Repos\fluent-bit\src\flb_config.c @ 139] 
06 00007ff7`4496af76 : 00000000`00000200 00000285`00000001 00000285`e1b6f220 00000285`e1b6f2e8 : fluent_bit!flb_config_exit+0x199 [D:\Devel\Repos\fluent-bit\src\flb_config.c @ 264] 
07 00007ff7`4495b033 : 00000285`e1b6f220 00000285`e1af77c0 00007ff7`44dc2ce8 00007ff7`44cc7830 : fluent_bit!flb_output_init_all+0xd6 [D:\Devel\Repos\fluent-bit\src\flb_output.c @ 605] 
08 00007ff7`44c158a8 : 00000000`00000003 00000285`e1b5fe30 00000000`00000000 00000000`00000000 : fluent_bit!flb_env_create+0x3 [D:\Devel\Repos\fluent-bit\src\flb_env.c @ 71] 
09 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!aes_gen_tables+0x408 [D:\Devel\Repos\fluent-bit\lib\mbedtls-2.16.5\library\aes.c @ 471] 
0a 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
0b 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   1  Id: e38.1bac Suspend: 0 Teb: 000000c4`fb777000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d0218a3f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`d021893e : 01d642d5`b413dcdb 00007ffd`d30f1e26 00000000`00000000 00000285`e1af5910 : KERNELBASE!WaitForMultipleObjectsEx+0xef
02 00000000`6dfba6d4 : 00000000`00000000 00000285`e1af5910 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjects+0xe
03 00000000`6df97a87 : 00000000`00000000 00000285`e1af5910 00000000`00000000 00000000`00000000 : tmmon64+0xaa6d4
04 00000000`6df97c2e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87a87
05 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87c2e
06 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
07 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   2  Id: e38.10f4 Suspend: 0 Teb: 000000c4`fb779000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d0218a3f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`d021893e : 00000285`e1b5fa60 00000000`001f0003 00000000`00000000 00000285`e1b5fa60 : KERNELBASE!WaitForMultipleObjectsEx+0xef
02 00007ffd`c9be987e : 00000000`00000000 00000000`00000000 00000000`00000000 00007ffd`c9be8637 : KERNELBASE!WaitForMultipleObjects+0xe
03 00007ffd`c9be9cfa : 00000000`00000002 000000c4`fbcff920 000000c4`fbcff908 00000000`000003e8 : TmUmEvt64!TmmonDestoryAddonObject+0x6b6e
04 00000000`6dfef046 : 00000000`00000000 00000000`00000128 00000000`00000000 00000000`00000000 : TmUmEvt64!TmmonDestoryAddonObject+0x6fea
05 00000000`6df97a87 : 00000285`e1af7b70 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0xdf046
06 00000000`6df97c2e : 00000000`6e021ec0 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87a87
07 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87c2e
08 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
09 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

#  3  Id: e38.12bc Suspend: 0 Teb: 000000c4`fb77b000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ff7`44c74f7d : 00000000`00000000 00000000`000001e4 000000c4`fbdfe560 00000000`00000000 : fluent_bit!get_banner_text+0x2 [minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 91] 
01 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!__acrt_mbs_to_wcs_cp::__l2::<lambda_7c9dea7b4ca7285d2cdb541a38da6275>::operator()+0x11 [minkernel\crts\ucrt\inc\corecrt_internal_win32_buffer.h @ 632] 
02 00007ff7`44c74d2f : 00000000`00001000 000000c4`fbdfe5e4 000000c4`fbdfe648 00000000`00000002 : fluent_bit!__acrt_convert_wcs_mbs_cp<char,wchar_t,<lambda_7c9dea7b4ca7285d2cdb541a38da6275>,__crt_win32_buffer_internal_dynamic_resizing>+0xb9 [minkernel\crts\ucrt\inc\corecrt_internal_win32_buffer.h @ 515] 
03 00007ff7`4495d486 : cccccccc`00000002 00000285`e1b87ed0 00000000`00000000 cccccccc`cccccccc : fluent_bit!__crt_seh_guarded_call<int>::operator()<<lambda_96b0dce3ad2a93c5c9ed548730f377b6>,<lambda_400f06e00cd32245fa85628833e911d2> &,<lambda_da1c5379cfd7439e775ba9665cd8bf87> >+0xbf
04 00007ff7`4495d5dd : 000000c4`fbdfe670 00000285`e1b85670 cccccccc`cccccccc cccccccc`00000000 : fluent_bit!gmtime_r+0x36 [D:\Devel\Repos\fluent-bit\include\fluent-bit\flb_compat.h @ 76] 
05 00007ff7`4495d76e : 00000000`000001d4 00000285`e1b85670 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!msgpack_pack_uint64+0x8d [D:\Devel\Repos\fluent-bit\lib\msgpack-3.2.0\include\msgpack\pack_template.h @ 350] 
06 00007ff7`4495c387 : 00000285`e1b85670 00000285`e1b84240 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!msgpack_pack_uint64+0x21e [D:\Devel\Repos\fluent-bit\lib\msgpack-3.2.0\include\msgpack\pack_template.h @ 350] 
07 00007ff7`44bed904 : 00000285`e1b84240 00000000`00000000 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!flb_hash_create+0x97 [D:\Devel\Repos\fluent-bit\src\flb_hash.c @ 123] 
08 00007ff7`44c58b90 : 00000285`e1b8f420 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!ecp_modp+0x134 [D:\Devel\Repos\fluent-bit\lib\mbedtls-2.16.5\library\ecp.c @ 1024] 
09 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!__crt_char_traits<char>::puttc_nolock+0xf [minkernel\crts\ucrt\inc\corecrt_internal_traits.h @ 109] 
0a (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!__crt_stdio_output::stream_output_adapter<char>::write_character_without_count_update+0x2f [minkernel\crts\ucrt\inc\corecrt_internal_stdio_output.h @ 164] 
0b (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : fluent_bit!__crt_stdio_output::output_adapter_common<char,__crt_stdio_output::stream_output_adapter<char> >::write_character+0x2f [minkernel\crts\ucrt\inc\corecrt_internal_stdio_output.h @ 61] 
0c 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!__crt_stdio_output::output_processor<char,__crt_stdio_output::stream_output_adapter<char>,__crt_stdio_output::format_validation_base<char,__crt_stdio_output::stream_output_adapter<char> > >::state_case_normal_tchar+0x64 [minkernel\crts\ucrt\inc\corecrt_internal_stdio_output.h @ 1720] 
0d 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
0e 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   4  Id: e38.14cc Suspend: 0 Teb: 000000c4`fb77d000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`cedcfb19 : 00000000`00000000 00000000`00000000 00007ffd`cedc0000 00000000`00000000 : ntdll!NtRemoveIoCompletion+0x14
01 00007ffd`d07a84d4 : 00007ffd`cedc2c70 00000285`e1b78c20 00000000`00000000 00000000`00000000 : mswsock!SockAsyncThread+0x79
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   5  Id: e38.35c Suspend: 0 Teb: 000000c4`fb78f000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d30c9eae : 00000285`e1ae4848 00000285`e1ae4848 00000285`e1b5e298 00000000`00000001 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x76e
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   6  Id: e38.1b68 Suspend: 0 Teb: 000000c4`fb791000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d30c9eae : 00000285`e1ae4848 00000285`e1ae4848 00000285`e1b5e298 00000000`00000001 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x76e
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

gitfool on 15 Jun 2020

I'm not sure but I think the symbols file is for fluent-bit.dll (not fluent-bit.exe), which I why I had trouble loading it, and also why I suspect the stack frames look wonky.

~If so, then I'll try again when you can provide me with another build with symbols.~

I changed the build target from ALL_BUILD (META) to fluent-bit-bin (EXECUTABLE) and I think the symbols look related this time, so trying to capture another crash dump.

gitfool on 15 Jun 2020

Here we go:

fluent-bit.exe_200615_114651.zip (incl. fluent-bit.exe symbols!)

Looking around WinDbg:

Microsoft (R) Windows Debugger Version 10.0.18362.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [D:\Work\Support\FluentBit\fluent-bit.exe_200615_114651\fluent-bit.exe_200615_114651.dmp]
User Mini Dump File with Full Memory: Only application data is available

Comment: '
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION'
Symbol search path is: srv*
Executable search path is: 
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)
Machine Name:
Debug session time: Mon Jun 15 23:46:51.000 2020 (UTC + 12:00)
System Uptime: 3 days 22:47:01.146
Process Uptime: 0 days 2:12:45.000
.................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(874.1378): Access violation - code c0000005 (first/second chance not available)
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for fluent-bit.exe
fluent_bit!write_text_ansi_nolock+0x72:
00007ff6`dfaa4966 8a06            mov     al,byte ptr [rsi] ds:000000b4`8dc00000=??
0:003> .sympath srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
*** WARNING: Unable to verify checksum for fluent-bit.exe
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*d:\symbols*http://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
0:003> .reload /f
.*** WARNING: Unable to verify checksum for fluent-bit.exe
................................

************* Symbol Loading Error Summary **************
Module name            Error
TmUmEvt64              The system cannot find the file specified
tmmon64                The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:003> lm
start             end                 module name
00000000`6df10000 00000000`6e074000   tmmon64    (no symbols)           
00007ff6`df760000 00007ff6`dfc74000   fluent_bit C (private pdb symbols)  D:\Devel\Repos\fluent-bit\build\bin\Debug\fluent-bit.pdb
00007ffd`c0490000 00007ffd`c04a6000   NapiNSP    (pdb symbols)          d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffd`c06d0000 00007ffd`c06de000   winrnr     (pdb symbols)          d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffd`c9b60000 00007ffd`c9cdc000   TmUmEvt64   (export symbols)       TmUmEvt64.dll
00007ffd`ca6d0000 00007ffd`ca6da000   rasadhlp   (pdb symbols)          d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffd`cc370000 00007ffd`cc3da000   FWPUCLNT   (pdb symbols)          d:\symbols\fwpuclnt.pdb\76287BDA6CE44E9F888125A7A291D1A81\fwpuclnt.pdb
00007ffd`ccf50000 00007ffd`ccf68000   nlaapi     (pdb symbols)          d:\symbols\nlaapi.pdb\2A36CC2DEED5441E824317E2B1D3B2DB1\nlaapi.pdb
00007ffd`ceb00000 00007ffd`ceb38000   IPHLPAPI   (pdb symbols)          d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffd`ceb40000 00007ffd`cebe2000   dnsapi     (pdb symbols)          d:\symbols\dnsapi.pdb\D92889441F4D4247AE4783B49D90B96F1\dnsapi.pdb
00007ffd`cedc0000 00007ffd`cee1c000   mswsock    (pdb symbols)          d:\symbols\mswsock.pdb\3ECFE529FA4746D98F4633C33EE460691\mswsock.pdb
00007ffd`cf440000 00007ffd`cf46b000   bcrypt     (pdb symbols)          d:\symbols\bcrypt.pdb\CF5F9E7894F74E0EB9157BD02BF123471\bcrypt.pdb
00007ffd`cf550000 00007ffd`cf560000   msasn1     (pdb symbols)          d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffd`cf580000 00007ffd`cf58f000   kernel_appcore   (pdb symbols)          d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffd`cf5f0000 00007ffd`cf771000   gdi32full   (pdb symbols)          d:\symbols\gdi32full.pdb\8C947FDC7D2B41BBACEBAE54C37B264F1\gdi32full.pdb
00007ffd`cf780000 00007ffd`cf874000   ucrtbase   (pdb symbols)          d:\symbols\ucrtbase.pdb\327103177C994904BA5C2BBBE08EC3331\ucrtbase.pdb
00007ffd`d01b0000 00007ffd`d01ce000   win32u     (pdb symbols)          d:\symbols\win32u.pdb\D746BAF2F0C04D5E9781C9CC9B12A2ED1\win32u.pdb
00007ffd`d01d0000 00007ffd`d03ed000   KERNELBASE   (pdb symbols)          d:\symbols\kernelbase.pdb\29BF43663E254495A4032C1C183AF4051\kernelbase.pdb
00007ffd`d03f0000 00007ffd`d05b9000   crypt32    (pdb symbols)          d:\symbols\crypt32.pdb\1CFEB418E4044D2FBAE01F87728DBEDA1\crypt32.pdb
00007ffd`d05c0000 00007ffd`d062a000   bcryptPrimitives   (pdb symbols)          d:\symbols\bcryptprimitives.pdb\A62F3617FC244277ADD68320C00250E91\bcryptprimitives.pdb
00007ffd`d0630000 00007ffd`d0796000   user32     (pdb symbols)          d:\symbols\user32.pdb\1E4B4E72316D49B2B245B9BA4F43265C1\user32.pdb
00007ffd`d07a0000 00007ffd`d084c000   kernel32   (pdb symbols)          d:\symbols\kernel32.pdb\FCB817ED2B6841C18F932CB4593BE1B71\kernel32.pdb
00007ffd`d0b80000 00007ffd`d0ca1000   rpcrt4     (pdb symbols)          d:\symbols\rpcrt4.pdb\526C08E00099481C96895AADD9B06CF01\rpcrt4.pdb
00007ffd`d0cb0000 00007ffd`d0d1a000   ws2_32     (pdb symbols)          d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffd`d0d20000 00007ffd`d0d28000   nsi        (pdb symbols)          d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffd`d0e90000 00007ffd`d0ec4000   gdi32      (pdb symbols)          d:\symbols\gdi32.pdb\3EE4BC8FDF37432694027B3713C3EF1C1\gdi32.pdb
00007ffd`d0f70000 00007ffd`d1235000   combase    (private pdb symbols)  d:\symbols\combase.pdb\F9E53A74EB3C4281A0A00B5F4B29E0011\combase.pdb
00007ffd`d12a0000 00007ffd`d12f2000   shlwapi    (pdb symbols)          d:\symbols\shlwapi.pdb\FB0E36D3E9774E3082EEF8F877CAE8721\shlwapi.pdb
00007ffd`d13f0000 00007ffd`d1492000   advapi32   (pdb symbols)          d:\symbols\advapi32.pdb\D8A05A9AD2FD4C3185A4041390FE54011\advapi32.pdb
00007ffd`d2de0000 00007ffd`d2e39000   sechost    (pdb symbols)          d:\symbols\sechost.pdb\3EE0CC142F5246A2A0879E93A04D88341\sechost.pdb
00007ffd`d2ff0000 00007ffd`d308e000   msvcrt     (pdb symbols)          d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffd`d3090000 00007ffd`d3098000   psapi      (pdb symbols)          d:\symbols\psapi.pdb\AEDA310D7062406A986FCE25C506506C1\psapi.pdb
00007ffd`d30a0000 00007ffd`d3270000   ntdll      (pdb symbols)          d:\symbols\ntdll.pdb\7B14B2F89F2D46F5998EFC7AF0F6EE0B1\ntdll.pdb
0:003> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


KEY_VALUES_STRING: 1

    Key  : AV.Fault
    Value: Read

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 341221

    Key  : Timeline.Process.Start.DeltaSec
    Value: 7965


PROCESSES_ANALYSIS: 1

SERVICE_ANALYSIS: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1

Timeline: !analyze.Start
    Name: <blank>
    Time: 2020-06-15T18:53:06.342Z
    Diff: 25575342 mSec

Timeline: Dump.Current
    Name: <blank>
    Time: 2020-06-15T11:46:51.0Z
    Diff: 0 mSec

Timeline: Process.Start
    Name: <blank>
    Time: 2020-06-15T09:34:06.0Z
    Diff: 7965000 mSec

Timeline: OS.Boot
    Name: <blank>
    Time: 2020-06-11T12:59:50.0Z
    Diff: 341221000 mSec


DUMP_CLASS: 2

DUMP_QUALIFIER: 400

CONTEXT:  (.ecxr)
rax=000000b48dbfeb5f rbx=000000b48dbfea5b rcx=00007ff6dfc46480
rdx=0000000000000012 rsi=000000b48dc00000 rdi=000000b48dbfebc0
rip=00007ff6dfaa4966 rsp=000000b48dbfd720 rbp=000000b4c7e25171
 r8=000000b48dbfed08  r9=000000003a226469 r10=0000000000000002
r11=000000b48dbfeca0 r12=0000000000000002 r13=0000000000000000
r14=0000000000000370 r15=0000000000000012
iopl=0         nv up ei ng nz ac pe cy
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010291
fluent_bit!write_text_ansi_nolock+0x72:
00007ff6`dfaa4966 8a06            mov     al,byte ptr [rsi] ds:000000b4`8dc00000=??
Resetting default scope

FAULTING_IP: 
fluent_bit!write_text_ansi_nolock+72 [minkernel\crts\ucrt\src\appcrt\lowio\write.cpp @ 401]
00007ff6`dfaa4966 8a06            mov     al,byte ptr [rsi]

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff6dfaa4966 (fluent_bit!write_text_ansi_nolock+0x0000000000000072)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 000000b48dc00000
Attempt to read from address 000000b48dc00000

DEFAULT_BUCKET_ID:  INVALID_POINTER_READ

PROCESS_NAME:  fluent-bit.exe

FOLLOWUP_IP: 
fluent_bit!write_text_ansi_nolock+72 [minkernel\crts\ucrt\src\appcrt\lowio\write.cpp @ 401]
00007ff6`dfaa4966 8a06            mov     al,byte ptr [rsi]

READ_ADDRESS:  000000b48dc00000 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  000000b48dc00000

WATSON_BKT_PROCSTAMP:  5ee7400d

WATSON_BKT_MODULE:  fluent-bit.exe

WATSON_BKT_MODSTAMP:  5ee7400d

WATSON_BKT_MODOFFSET:  344966

BUILD_VERSION_STRING:  10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)

MODLIST_WITH_TSCHKSUM_HASH:  f62799efccfc96deadc6ed018d33e5a031a41d58

MODLIST_SHA1_HASH:  afc6ba214a6e2f01bd2716f01521ec8f38e36019

COMMENT:  
*** "C:\PharosSystems\SysinternalsSuite\procdump64.exe" -ma -e fluent-bit.exe
*** Unhandled exception: C0000005.ACCESS_VIOLATION

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

PRODUCT_TYPE:  3

SUITE_MASK:  400

DUMP_FLAGS:  8000c07

DUMP_TYPE:  3

ANALYSIS_SESSION_HOST:  SEANFAUSETT

ANALYSIS_SESSION_TIME:  06-16-2020 06:53:06.0342

ANALYSIS_VERSION: 10.0.18362.1 amd64fre

THREAD_ATTRIBUTES: 
OS_LOCALE:  ENU

BUGCHECK_STR:  APPLICATION_FAULT_INVALID_POINTER_READ

PRIMARY_PROBLEM_CLASS:  APPLICATION_FAULT

PROBLEM_CLASSES: 

    ID:     [0n313]
    Type:   [@ACCESS_VIOLATION]
    Class:  Addendum
    Scope:  BUCKET_ID
    Name:   Omit
    Data:   Omit
    PID:    [Unspecified]
    TID:    [0x1378]
    Frame:  [0] : fluent_bit!write_text_ansi_nolock

    ID:     [0n285]
    Type:   [INVALID_POINTER_READ]
    Class:  Primary
    Scope:  DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
            BUCKET_ID
    Name:   Add
    Data:   Omit
    PID:    [Unspecified]
    TID:    [0x1378]
    Frame:  [0] : fluent_bit!write_text_ansi_nolock

LAST_CONTROL_TRANSFER:  from 00007ff6dfaa4f7d to 00007ff6dfaa4966

STACK_TEXT:  
000000b4`8dbfd720 00007ff6`dfaa4f7d : 00000000`00000000 00000000`000001ec 000000b4`8dbfebf0 00000000`00000000 : fluent_bit!write_text_ansi_nolock+0x72
000000b4`8dbfeb90 00007ff6`dfaa4d2f : 00000000`00001000 000000b4`8dbfec74 000000b4`8dbfecd8 00000000`00000002 : fluent_bit!_write_nolock+0x20d
000000b4`8dbfec30 00007ff6`df78d486 : cccccccc`00000002 00000241`4c378870 00000000`00000000 cccccccc`cccccccc : fluent_bit!_write+0xab
000000b4`8dbfec80 00007ff6`df78d5dd : 000000b4`8dbfed00 00000241`4c3714e0 cccccccc`cccccccc cccccccc`00000000 : fluent_bit!log_push+0x56
000000b4`8dbfecc0 00007ff6`df78d76e : 00000000`000001dc 00000241`4c3714e0 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!log_read+0x8d
000000b4`8dbffd30 00007ff6`df78c387 : 00000241`4c3714e0 00000241`4c2da390 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!log_worker_collector+0x14e
000000b4`8dbffd90 00007ff6`dfa1d904 : 00000241`4c2da390 00000000`00000000 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!step_callback+0x77
000000b4`8dbffdd0 00007ff6`dfa88b90 : 00000241`4c380a60 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!pthread_create_wrapper+0x74
000000b4`8dbffe10 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!thread_start<unsigned int (__cdecl*)(void *),1>+0x50
000000b4`8dbffe40 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
000000b4`8dbffe70 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


THREAD_SHA1_HASH_MOD_FUNC:  b5cdc71de2b39e2787960decf654963447efae48

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  1e9fb1740fa6badb047a80c34ff663edcdbc750c

THREAD_SHA1_HASH_MOD:  86e11f2cb1bfae317fbe4a25f4030f00f1d95f28

FAULT_INSTR_CODE:  ff48068a

FAULTING_SOURCE_LINE:  minkernel\crts\ucrt\src\appcrt\lowio\write.cpp

FAULTING_SOURCE_FILE:  minkernel\crts\ucrt\src\appcrt\lowio\write.cpp

FAULTING_SOURCE_LINE_NUMBER:  401

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  fluent_bit!write_text_ansi_nolock+72

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: fluent_bit

IMAGE_NAME:  fluent-bit.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  5ee7400d

STACK_COMMAND:  dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~3s ; .ecxr ; kb

FAILURE_BUCKET_ID:  INVALID_POINTER_READ_c0000005_fluent-bit.exe!write_text_ansi_nolock

BUCKET_ID:  APPLICATION_FAULT_INVALID_POINTER_READ_fluent_bit!write_text_ansi_nolock+72

FAILURE_EXCEPTION_CODE:  c0000005

FAILURE_IMAGE_NAME:  fluent-bit.exe

BUCKET_ID_IMAGE_STR:  fluent-bit.exe

FAILURE_MODULE_NAME:  fluent_bit

BUCKET_ID_MODULE_STR:  fluent_bit

FAILURE_FUNCTION_NAME:  write_text_ansi_nolock

BUCKET_ID_FUNCTION_STR:  write_text_ansi_nolock

BUCKET_ID_OFFSET:  72

BUCKET_ID_MODTIMEDATESTAMP:  5ee7400d

BUCKET_ID_MODCHECKSUM:  0

BUCKET_ID_MODVER_STR:  0.0.0.0

BUCKET_ID_PREFIX_STR:  APPLICATION_FAULT_INVALID_POINTER_READ_

FAILURE_PROBLEM_CLASS:  APPLICATION_FAULT

FAILURE_SYMBOL_NAME:  fluent-bit.exe!write_text_ansi_nolock

TARGET_TIME:  2020-06-15T11:46:51.000Z

OSBUILD:  14393

OSSERVICEPACK:  2969

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 Server TerminalServer DataCenter SingleUserTS

USER_LCID:  0

OSBUILD_TIMESTAMP:  2019-05-04 16:40:41

BUILDDATESTAMP_STR:  180806-1810

BUILDLAB_STR:  rs1_release_inmarket_aim

BUILDOSVER_STR:  10.0.14393.2430

ANALYSIS_SESSION_ELAPSED_TIME:  46b

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:invalid_pointer_read_c0000005_fluent-bit.exe!write_text_ansi_nolock

FAILURE_ID_HASH:  {56f7cd40-05c4-0988-7a3c-4ce9bf04ba35}

Followup:     MachineOwner
---------

0:003> ~*kb

   0  Id: 874.db4 Suspend: 0 Teb: 000000b4`8d511000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`cedcc268 : 00000000`00000000 00000000`00000000 00000241`4c3c8910 00007ffd`cedccc3a : ntdll!NtDeviceIoControlFile+0x14
01 00007ffd`d0cbc687 : 00000000`0000020c 00000241`4c370480 00000000`00000000 000000b5`c4cff914 : mswsock!WSPSend+0x1c8
02 00007ff6`df7dcf52 : 00000000`00000000 00000000`00000000 00000000`00000000 cccccccc`cccccccc : ws2_32!send+0x167
03 00007ff6`df7dcfda : cccccccc`00000001 00000241`4c3c3c20 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!flb_output_return+0xa2 [D:\Devel\Repos\fluent-bit\include\fluent-bit\flb_output.h @ 467] 
04 00007ff6`df7df1d4 : 00000241`00000001 00000000`00000000 00000000`00000000 00007ff6`dfbffb38 : fluent_bit!flb_output_return_do+0x3a [D:\Devel\Repos\fluent-bit\include\fluent-bit\flb_output.h @ 503] 
05 00007ff6`df7accf2 : 00000241`4d0900a0 00000000`00001ac4 00000241`4c2e7f20 cccccccc`00000032 : fluent_bit!cb_es_flush+0x774 [D:\Devel\Repos\fluent-bit\plugins\out_es\es.c @ 703] 
06 00007ff6`df8c20e2 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!output_pre_cb_flush+0x142 [D:\Devel\Repos\fluent-bit\include\fluent-bit\flb_output.h @ 372] 
07 00007ffd`d023e7b1 : 00007ff6`df7acbb0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0x22 [D:\Devel\Repos\fluent-bit\lib\flb_libco\fiber.c @ 23] 
08 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21

   1  Id: 874.19f0 Suspend: 0 Teb: 000000b4`8d517000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d0218a3f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`d021893e : 01d642f8`40cd6460 00007ffd`d30f1e26 00000000`00000000 00000241`4c2e5e30 : KERNELBASE!WaitForMultipleObjectsEx+0xef
02 00000000`6dfba6d4 : 00000000`00000000 00000241`4c2e5e30 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjects+0xe
03 00000000`6df97a87 : 00000000`00000000 00000241`4c2e5e30 00000000`00000000 00000000`00000000 : tmmon64+0xaa6d4
04 00000000`6df97c2e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87a87
05 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87c2e
06 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
07 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   2  Id: 874.13fc Suspend: 0 Teb: 000000b4`8d519000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d0218a3f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`d021893e : 00000241`4c34fac0 00000000`001f0003 00000000`00000000 00000241`4c34fac0 : KERNELBASE!WaitForMultipleObjectsEx+0xef
02 00007ffd`c9be987e : 00000000`00000000 00000000`00000000 00000000`00000000 00007ffd`c9be8637 : KERNELBASE!WaitForMultipleObjects+0xe
03 00007ffd`c9be9cfa : 00000000`00000002 000000b4`8daffb40 000000b4`8daffb28 00000000`000003e8 : TmUmEvt64!TmmonDestoryAddonObject+0x6b6e
04 00000000`6dfef046 : 00000000`00000000 00000000`00000128 00000000`00000000 00000000`00000000 : TmUmEvt64!TmmonDestoryAddonObject+0x6fea
05 00000000`6df97a87 : 00000241`4c2dfb90 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0xdf046
06 00000000`6df97c2e : 00000000`6e021ec0 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87a87
07 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87c2e
08 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
09 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

#  3  Id: 874.1378 Suspend: 0 Teb: 000000b4`8d51b000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ff6`dfaa4f7d : 00000000`00000000 00000000`000001ec 000000b4`8dbfebf0 00000000`00000000 : fluent_bit!write_text_ansi_nolock+0x72 [minkernel\crts\ucrt\src\appcrt\lowio\write.cpp @ 401] 
01 00007ff6`dfaa4d2f : 00000000`00001000 000000b4`8dbfec74 000000b4`8dbfecd8 00000000`00000002 : fluent_bit!_write_nolock+0x20d [minkernel\crts\ucrt\src\appcrt\lowio\write.cpp @ 679] 
02 00007ff6`df78d486 : cccccccc`00000002 00000241`4c378870 00000000`00000000 cccccccc`cccccccc : fluent_bit!_write+0xab [minkernel\crts\ucrt\src\appcrt\lowio\write.cpp @ 64] 
03 00007ff6`df78d5dd : 000000b4`8dbfed00 00000241`4c3714e0 cccccccc`cccccccc cccccccc`00000000 : fluent_bit!log_push+0x56 [D:\Devel\Repos\fluent-bit\src\flb_log.c @ 66] 
04 00007ff6`df78d76e : 00000000`000001dc 00000241`4c3714e0 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!log_read+0x8d [D:\Devel\Repos\fluent-bit\src\flb_log.c @ 99] 
05 00007ff6`df78c387 : 00000241`4c3714e0 00000241`4c2da390 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!log_worker_collector+0x14e [D:\Devel\Repos\fluent-bit\src\flb_log.c @ 123] 
06 00007ff6`dfa1d904 : 00000241`4c2da390 00000000`00000000 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!step_callback+0x77 [D:\Devel\Repos\fluent-bit\src\flb_worker.c @ 54] 
07 00007ff6`dfa88b90 : 00000241`4c380a60 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!pthread_create_wrapper+0x74 [D:\Devel\Repos\fluent-bit\lib\monkey\mk_core\external\winpthreads.c @ 654] 
08 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!thread_start<unsigned int (__cdecl*)(void *),1>+0x50 [minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 97] 
09 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
0a 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   4  Id: 874.1578 Suspend: 0 Teb: 000000b4`8d51d000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`cedcfb19 : 00000000`00000000 00000000`00000000 00007ffd`cedc0000 00000000`00000000 : ntdll!NtRemoveIoCompletion+0x14
01 00007ffd`d07a84d4 : 00007ffd`cedc2c70 00000241`4c3695d0 00000000`00000000 00000000`00000000 : mswsock!SockAsyncThread+0x79
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   5  Id: 874.18a4 Suspend: 0 Teb: 000000b4`8d52d000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d30c9eae : 00000241`4c2d5778 00000241`4c2d5778 00000241`4c34e758 00000000`00000001 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x76e
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   6  Id: 874.11e4 Suspend: 0 Teb: 000000b4`8d52f000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d30c9eae : 00000241`4c2d5778 00000241`4c2d5778 00000241`4c34e758 00000000`00000001 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x76e
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
0:003> dd 000000b48dc00000 
000000b4`8dc00000  ???????? ???????? ???????? ????????
000000b4`8dc00010  ???????? ???????? ???????? ????????
000000b4`8dc00020  ???????? ???????? ???????? ????????
000000b4`8dc00030  ???????? ???????? ???????? ????????
000000b4`8dc00040  ???????? ???????? ???????? ????????
000000b4`8dc00050  ???????? ???????? ???????? ????????
000000b4`8dc00060  ???????? ???????? ???????? ????????
000000b4`8dc00070  ???????? ???????? ???????? ????????
0:003> dx Debugger.Sessions[0].Processes[2164].Threads[4984].Stack.Frames[4].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[2164].Threads[4984].Stack.Frames[4].SwitchTo()
000000b4`8dbffd30 int64 fd = 0n476
000000b4`8dbffd38 struct flb_log * log = 0x00000241`4c3714e0
000000b4`8dbfece0 int bytes = 0n2636
000000b4`8dbfed00 struct log_message msg = struct log_message
0:003> dx -r1 (*((fluent_bit!log_message *)0xb48dbfed00))
(*((fluent_bit!log_message *)0xb48dbfed00))                 [Type: log_message]
    [+0x000] size             : 0x626d79223a226469 [Type: unsigned __int64]
    [+0x008] msg              : "Nt3IBpWtYKTqntKtI","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":137096,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.15","_type":"_doc","_id":"y2bNt3IBpWtYKTqntKtI","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":136542,"_primary_term":1,"status":201}}]}." [Type: char [4088]]
0:003> db 000000b4`8dbfed00 L0n2636
000000b4`8dbfed00  69 64 22 3a 22 79 6d 62-4e 74 33 49 42 70 57 74  id":"ymbNt3IBpWt
000000b4`8dbfed10  59 4b 54 71 6e 74 4b 74-49 22 2c 22 5f 76 65 72  YKTqntKtI","_ver
000000b4`8dbfed20  73 69 6f 6e 22 3a 31 2c-22 72 65 73 75 6c 74 22  sion":1,"result"
000000b4`8dbfed30  3a 22 63 72 65 61 74 65-64 22 2c 22 5f 73 68 61  :"created","_sha
000000b4`8dbfed40  72 64 73 22 3a 7b 22 74-6f 74 61 6c 22 3a 32 2c  rds":{"total":2,
000000b4`8dbfed50  22 73 75 63 63 65 73 73-66 75 6c 22 3a 32 2c 22  "successful":2,"
000000b4`8dbfed60  66 61 69 6c 65 64 22 3a-30 7d 2c 22 5f 73 65 71  failed":0},"_seq
000000b4`8dbfed70  5f 6e 6f 22 3a 31 33 37-30 39 36 2c 22 5f 70 72  _no":137096,"_pr
000000b4`8dbfed80  69 6d 61 72 79 5f 74 65-72 6d 22 3a 31 2c 22 73  imary_term":1,"s
000000b4`8dbfed90  74 61 74 75 73 22 3a 32-30 31 7d 7d 2c 7b 22 69  tatus":201}},{"i
000000b4`8dbfeda0  6e 64 65 78 22 3a 7b 22-5f 69 6e 64 65 78 22 3a  ndex":{"_index":
000000b4`8dbfedb0  22 6c 6f 67 73 74 61 73-68 2d 32 30 32 30 2e 30  "logstash-2020.0
000000b4`8dbfedc0  36 2e 31 35 22 2c 22 5f-74 79 70 65 22 3a 22 5f  6.15","_type":"_
000000b4`8dbfedd0  64 6f 63 22 2c 22 5f 69-64 22 3a 22 79 32 62 4e  doc","_id":"y2bN
000000b4`8dbfede0  74 33 49 42 70 57 74 59-4b 54 71 6e 74 4b 74 49  t3IBpWtYKTqntKtI
000000b4`8dbfedf0  22 2c 22 5f 76 65 72 73-69 6f 6e 22 3a 31 2c 22  ","_version":1,"
000000b4`8dbfee00  72 65 73 75 6c 74 22 3a-22 63 72 65 61 74 65 64  result":"created
000000b4`8dbfee10  22 2c 22 5f 73 68 61 72-64 73 22 3a 7b 22 74 6f  ","_shards":{"to
000000b4`8dbfee20  74 61 6c 22 3a 32 2c 22-73 75 63 63 65 73 73 66  tal":2,"successf
000000b4`8dbfee30  75 6c 22 3a 32 2c 22 66-61 69 6c 65 64 22 3a 30  ul":2,"failed":0
000000b4`8dbfee40  7d 2c 22 5f 73 65 71 5f-6e 6f 22 3a 31 33 36 35  },"_seq_no":1365
000000b4`8dbfee50  34 32 2c 22 5f 70 72 69-6d 61 72 79 5f 74 65 72  42,"_primary_ter
000000b4`8dbfee60  6d 22 3a 31 2c 22 73 74-61 74 75 73 22 3a 32 30  m":1,"status":20
000000b4`8dbfee70  31 7d 7d 5d 7d 0a 00 00-00 00 00 00 00 00 00 00  1}}]}...........
000000b4`8dbfee80  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfee90  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeea0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeeb0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeec0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeed0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeee0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeef0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef00  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef10  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef20  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef30  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef40  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef50  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef60  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef70  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef80  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfef90  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfefa0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfefb0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfefc0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfefd0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfefe0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbfeff0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff000  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff010  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff020  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff030  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff040  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff050  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff060  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff070  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff080  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff090  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff0a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff0b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff0c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff0d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff0e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff0f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff100  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff110  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff120  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff130  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff140  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff150  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff160  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff170  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff180  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff190  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff1a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff1b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff1c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff1d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff1e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff1f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff200  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff210  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff220  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff230  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff240  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff250  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff260  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff270  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff280  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff290  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff2a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff2b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff2c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff2d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff2e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff2f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff300  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff310  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff320  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff330  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff340  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff350  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff360  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff370  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff380  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff390  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff3a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff3b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff3c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff3d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff3e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff3f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff400  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff410  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff420  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff430  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff440  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff450  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff460  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff470  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff480  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff490  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff4a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff4b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff4c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff4d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff4e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff4f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff500  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff510  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff520  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff530  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff540  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff550  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff560  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff570  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff580  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff590  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff5a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff5b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff5c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff5d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff5e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff5f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff600  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff610  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff620  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff630  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff640  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff650  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff660  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff670  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff680  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff690  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff6a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff6b0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff6c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff6d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff6e0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff6f0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff700  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff710  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff720  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff730  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000b4`8dbff740  00 00 00 00 00 00 00 00-00 00 00 00              ............

The evidence indicates that log_read is not returning a log_message struct in this case, since the size field contains msg characters, so when the bogus size is used in log_push it reads past the buffer.

gitfool on 15 Jun 2020

@gitfool I preapred a branch win32-2251 for your testing.
It's basically a set of small Windows patches on top of v1.4.6.

Commit: https://github.com/fluent/fluent-bit/commits/win32-2251
AppVeyor Build: td-agent-bit-1.4.6-win64 (AppVeyor)

The testing package is available from the second link; It includes a
correct PDB file, so you can readily test things out.

I'm not sure but I think the symbols file is for fluent-bit.dll (not
fluent-bit.exe), which I why I had trouble loading it, and also why I
suspect the stack frames look wonky.

That's a good catch. I took it account into f1810e2d and made it sure
that both EXE and DLL have separate PDB files.

The branch win32-2251 includes this fix, and it now produces the
following file assets:

build/bin/
├── fluent-bit.dll
├── fluent-bit.dll.pdb
├── fluent-bit.exe
└── fluent-bit.pdb

Please feel free to tell me if you notice anything.

fujimotos on 16 Jun 2020

👍1

... and I just notice you already did get the dump in the https://github.com/fluent/fluent-bit/issues/2251#issuecomment-644327972.

I'll take a closer look at your log tomorrow and investigate the root
cause of the crash issue.

fujimotos on 16 Jun 2020

👍1

A couple of questions...

Why is there a shared library build of fluent-bit.dll anyway?
- I would've thought fluent-bit.exe is what most, if not all, people would use (as a service)
Why is it building with Visual Studio 2017 instead of Visual Studio 2019?

gitfool on 16 Jun 2020

@gitfool I pushed a fix to the branch win32-2251.

Branch HEAD: 2050d20
AppVeyor Build: builds/33564849

I added a safety check that prevents log_push() from reading over
the struct boundary. So it should not cause segfault even when it
receives a corrupted message.

I hope this helps you.

Why is there a shared library build of fluent-bit.dll anyway?

This is mostly for customary reasons.

Fluent Bit is designed to be embeddable into other C programs.
Due to this requirement, every distribution package (including
Linux, Windows and MacOS) contains a shared library.

That being said, I concur that the Windows users ralely make use
of that "embeddable" part of Fluent Bit.

I also think that there might be a better form of distrubution
that suits the current usage pattern (e.g. distributing only
fluent-bit.exe, so Windows users can use it without expanding).

Why is it building with Visual Studio 2017 instead of Visual Studio 2019

When we started to port Fluent Bit to Windows in 2018, the
latest available Visual Studio was 2017.

In fact, both VS2017 and VS2019 can compile Fluent Bit just
fine, so either is okay.

fujimotos on 17 Jun 2020

@fujimotos I don't think that's a good change. The read using socket recv is already limited to sizeof(struct log_message). This buffer will not be overrun.

Also, when it does fail, bytes seems to always be 2636. (See dump analysis above.)

My gut feel is the root cause is due to the reads being split in rare cases. In such a case the struct log_message interpretation of the buffer is invalid as the subsequent read only contains more msg characters. I've been trying to prove it with some debug logging. but adding logging seems to reduce the rate of failure to the point that it didn't happen overnight.

gitfool on 17 Jun 2020

@gitfool I realized I pushed a wrong commit. I re-updated the
HEAD with the correct one.

HEAD: d38a6f9c
AppVeyor: build/33566040

My gut feel is the root cause is due to the reads being split in rare cases.

My guess is the same. If read(2) returns only the initial a few bytes
(or read corrupted message via concurrent writes or reading msgs
not from the begenning...), msg->size can contain invalid size.

The root issue is that the current logic assumes atomic write/read,
but it is not really something guaranteed in the POSIX standard.

We really should sort out the logging logic, but this fix should at
least prevent it from crashing.

fujimotos on 17 Jun 2020

@fujimotos okay, that will avoid the crash, although the pointer deref shouldn't be needed, and wouldn't it be useful to add a call to perror to track failures?

if (bytes <= 0) {
    perror("flb_pipe_r");
    return -1;
}
if (msg.size > sizeof(msg.msg)) {
    perror("flb_pipe_r msg.size");
    return -1;
}

Would still be good to fix the root cause though.

gitfool on 17 Jun 2020

@fujimotos I finally have proof that the it's due to a split read. I was using the following change:

if (msg.size > sizeof(msg.msg)) {
    DebugBreak();
    perror("flb_pipe_r msg.size");
    return -1;
}

Attached to process with WinDbg and waited until it broke into the debugger:

Microsoft (R) Windows Debugger Version 10.0.19041.1 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [D:\Work\Support\FluentBit\fluent-bit_1684_2020-06-17_03-16-12-414_0770\fluent-bit_1684_2020-06-17_03-16-12-414_0770.dmp]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 10 Version 14393 MP (2 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
14393.3564.amd64fre.rs1_release.200303-1942
Machine Name:
Debug session time: Wed Jun 17 15:16:12.000 2020 (UTC + 12:00)
System Uptime: 5 days 14:16:20.724
Process Uptime: 0 days 0:12:12.000
.................................
This dump file has a breakpoint exception stored in it.
The stored exception information can be accessed via .ecxr.
For analysis of this file, run !analyze -v
*** WARNING: Unable to verify checksum for fluent-bit.exe
KERNELBASE!DebugBreak+0x2:
00007ffd`d0295f72 cc              int     3
0:003> .sympath+ D:\Work\Support\FluentBit\fluent-bit_1684_2020-06-17_03-16-12-414_0770
Symbol search path is: srv*D:\Symbols*http://msdl.microsoft.com/download/symbols;D:\Work\Support\FluentBit\fluent-bit_1684_2020-06-17_03-16-12-414_0770
Expanded Symbol search path is: srv*d:\symbols*http://msdl.microsoft.com/download/symbols;d:\work\support\fluentbit\fluent-bit_1684_2020-06-17_03-16-12-414_0770

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*D:\Symbols*http://msdl.microsoft.com/download/symbols
OK                                             D:\Work\Support\FluentBit\fluent-bit_1684_2020-06-17_03-16-12-414_0770
0:003> .reload /f
.*** WARNING: Unable to verify checksum for fluent-bit.exe
................................

************* Symbol Loading Error Summary **************
Module name            Error
TmUmEvt64              The system cannot find the file specified
tmmon64                The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:003> lm
start             end                 module name
00000000`6df10000 00000000`6e074000   tmmon64    (no symbols)           
00007ff6`32a20000 00007ff6`32f34000   fluent_bit C (private pdb symbols)  d:\work\support\fluentbit\fluent-bit_1684_2020-06-17_03-16-12-414_0770\fluent-bit.pdb
00007ffd`c0490000 00007ffd`c04a6000   NapiNSP    (pdb symbols)          d:\symbols\NapiNSP.pdb\6A1E2DD884D24B0B8F89441A8693673C1\NapiNSP.pdb
00007ffd`c06d0000 00007ffd`c06de000   winrnr     (pdb symbols)          d:\symbols\winrnr.pdb\A497F69E1B7A41F28EF8390F833AB47A1\winrnr.pdb
00007ffd`c9b60000 00007ffd`c9cdc000   TmUmEvt64   (export symbols)       TmUmEvt64.dll
00007ffd`ca6d0000 00007ffd`ca6da000   rasadhlp   (pdb symbols)          d:\symbols\rasadhlp.pdb\5EB493878A6F4865AF6EE51C56207CF01\rasadhlp.pdb
00007ffd`cc370000 00007ffd`cc3da000   FWPUCLNT   (pdb symbols)          d:\symbols\fwpuclnt.pdb\76287BDA6CE44E9F888125A7A291D1A81\fwpuclnt.pdb
00007ffd`ccf50000 00007ffd`ccf68000   nlaapi     (pdb symbols)          d:\symbols\nlaapi.pdb\2A36CC2DEED5441E824317E2B1D3B2DB1\nlaapi.pdb
00007ffd`ceb00000 00007ffd`ceb38000   IPHLPAPI   (pdb symbols)          d:\symbols\iphlpapi.pdb\4C53537F0DAA44AF97FA62AD31BC46481\iphlpapi.pdb
00007ffd`ceb40000 00007ffd`cebe2000   dnsapi     (pdb symbols)          d:\symbols\dnsapi.pdb\D92889441F4D4247AE4783B49D90B96F1\dnsapi.pdb
00007ffd`cedc0000 00007ffd`cee1c000   mswsock    (pdb symbols)          d:\symbols\mswsock.pdb\3ECFE529FA4746D98F4633C33EE460691\mswsock.pdb
00007ffd`cf440000 00007ffd`cf46b000   bcrypt     (pdb symbols)          d:\symbols\bcrypt.pdb\CF5F9E7894F74E0EB9157BD02BF123471\bcrypt.pdb
00007ffd`cf550000 00007ffd`cf560000   msasn1     (pdb symbols)          d:\symbols\msasn1.pdb\960B2EDD42C2489F8C1AB3816BF8E9341\msasn1.pdb
00007ffd`cf580000 00007ffd`cf58f000   kernel_appcore   (pdb symbols)          d:\symbols\Kernel.Appcore.pdb\34F749B066CB4C2DA6F3E8EEB6013A8A1\Kernel.Appcore.pdb
00007ffd`cf5f0000 00007ffd`cf771000   gdi32full   (pdb symbols)          d:\symbols\gdi32full.pdb\8C947FDC7D2B41BBACEBAE54C37B264F1\gdi32full.pdb
00007ffd`cf780000 00007ffd`cf874000   ucrtbase   (pdb symbols)          d:\symbols\ucrtbase.pdb\327103177C994904BA5C2BBBE08EC3331\ucrtbase.pdb
00007ffd`d01b0000 00007ffd`d01ce000   win32u     (pdb symbols)          d:\symbols\win32u.pdb\D746BAF2F0C04D5E9781C9CC9B12A2ED1\win32u.pdb
00007ffd`d01d0000 00007ffd`d03ed000   KERNELBASE   (pdb symbols)          d:\symbols\kernelbase.pdb\29BF43663E254495A4032C1C183AF4051\kernelbase.pdb
00007ffd`d03f0000 00007ffd`d05b9000   crypt32    (pdb symbols)          d:\symbols\crypt32.pdb\1CFEB418E4044D2FBAE01F87728DBEDA1\crypt32.pdb
00007ffd`d05c0000 00007ffd`d062a000   bcryptPrimitives   (pdb symbols)          d:\symbols\bcryptprimitives.pdb\A62F3617FC244277ADD68320C00250E91\bcryptprimitives.pdb
00007ffd`d0630000 00007ffd`d0796000   user32     (pdb symbols)          d:\symbols\user32.pdb\1E4B4E72316D49B2B245B9BA4F43265C1\user32.pdb
00007ffd`d07a0000 00007ffd`d084c000   kernel32   (pdb symbols)          d:\symbols\kernel32.pdb\FCB817ED2B6841C18F932CB4593BE1B71\kernel32.pdb
00007ffd`d0b80000 00007ffd`d0ca1000   rpcrt4     (pdb symbols)          d:\symbols\rpcrt4.pdb\526C08E00099481C96895AADD9B06CF01\rpcrt4.pdb
00007ffd`d0cb0000 00007ffd`d0d1a000   ws2_32     (pdb symbols)          d:\symbols\ws2_32.pdb\88345C10081E4DA7BC9942E4DDCBA06C1\ws2_32.pdb
00007ffd`d0d20000 00007ffd`d0d28000   nsi        (pdb symbols)          d:\symbols\nsi.pdb\8D9A83E03AC843A09A4D7329EB9186C61\nsi.pdb
00007ffd`d0e90000 00007ffd`d0ec4000   gdi32      (pdb symbols)          d:\symbols\gdi32.pdb\3EE4BC8FDF37432694027B3713C3EF1C1\gdi32.pdb
00007ffd`d0f70000 00007ffd`d1235000   combase    (private pdb symbols)  d:\symbols\combase.pdb\F9E53A74EB3C4281A0A00B5F4B29E0011\combase.pdb
00007ffd`d12a0000 00007ffd`d12f2000   shlwapi    (pdb symbols)          d:\symbols\shlwapi.pdb\FB0E36D3E9774E3082EEF8F877CAE8721\shlwapi.pdb
00007ffd`d13f0000 00007ffd`d1492000   advapi32   (pdb symbols)          d:\symbols\advapi32.pdb\D8A05A9AD2FD4C3185A4041390FE54011\advapi32.pdb
00007ffd`d2de0000 00007ffd`d2e39000   sechost    (pdb symbols)          d:\symbols\sechost.pdb\3EE0CC142F5246A2A0879E93A04D88341\sechost.pdb
00007ffd`d2ff0000 00007ffd`d308e000   msvcrt     (pdb symbols)          d:\symbols\msvcrt.pdb\BAFC8C0A0A7648E0A6C2FC97B72A04B11\msvcrt.pdb
00007ffd`d3090000 00007ffd`d3098000   psapi      (pdb symbols)          d:\symbols\psapi.pdb\AEDA310D7062406A986FCE25C506506C1\psapi.pdb
00007ffd`d30a0000 00007ffd`d3270000   ntdll      (pdb symbols)          d:\symbols\ntdll.pdb\7B14B2F89F2D46F5998EFC7AF0F6EE0B1\ntdll.pdb
0:003> ~*kb

   0  Id: 770.138c Suspend: 1 Teb: 000000a3`0ba63000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`cedccbd0 : 00000235`23536a50 00007ffd`d0cbadf0 00000000`00000000 00007ff6`32d662b8 : ntdll!NtClose+0x14
01 00007ffd`cedcc916 : cccccccc`cccccccc 00000235`23536a50 00000000`00000000 cccccccc`cccccccc : mswsock!SockCloseSocket+0x200
02 00007ffd`d0cbb83e : 00000000`00000000 00000235`234578e0 00000235`234497c0 00000000`000002cc : mswsock!WSPCloseSocket+0x86
03 00007ff6`32ce39e9 : 00000000`00000000 cccccccc`00000000 cccccccc`cccccccc cccccccc`cccccccc : ws2_32!closesocket+0x11e
04 00007ff6`32a6c6b7 : 00000000`000002cc 00000235`234fba70 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!evutil_closesocket+0x29 [D:\Devel\Repos\fluent-bit\lib\monkey\mk_core\deps\libevent\evutil.c @ 445] 
05 00007ff6`32a6c30c : 00000235`234fba70 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!destroy_conn+0x97 [D:\Devel\Repos\fluent-bit\src\flb_upstream.c @ 196] 
06 00007ff6`32a9f1d2 : 00000235`234fba70 00000000`00000000 00000000`00000000 00007ff6`32ebfb48 : fluent_bit!flb_upstream_conn_release+0x24c [D:\Devel\Repos\fluent-bit\src\flb_upstream.c @ 361] 
07 00007ff6`32a6cd12 : 00000235`241c5f40 00000000`000030a7 00000235`234578a0 cccccccc`00000032 : fluent_bit!cb_es_flush+0x752 [D:\Devel\Repos\fluent-bit\plugins\out_es\es.c @ 700] 
08 00007ff6`32b82102 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!output_pre_cb_flush+0x142 [D:\Devel\Repos\fluent-bit\include\fluent-bit\flb_output.h @ 372] 
09 00007ffd`d023e7b1 : 00007ff6`32a6cbd0 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!co_thunk+0x22 [D:\Devel\Repos\fluent-bit\lib\flb_libco\fiber.c @ 23] 
0a 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!BaseFiberStart+0x21

   1  Id: 770.e18 Suspend: 1 Teb: 000000a3`0ba69000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d0218a3f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`d021893e : 01d64454`16966536 00007ffd`d30f1e26 00000000`00000000 00000235`23455e30 : KERNELBASE!WaitForMultipleObjectsEx+0xef
02 00000000`6dfba6d4 : 00000000`00000000 00000235`23455e30 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjects+0xe
03 00000000`6df97a87 : 00000000`00000000 00000235`23455e30 00000000`00000000 00000000`00000000 : tmmon64+0xaa6d4
04 00000000`6df97c2e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87a87
05 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87c2e
06 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
07 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   2  Id: 770.1284 Suspend: 1 Teb: 000000a3`0ba6b000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d0218a3f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0x14
01 00007ffd`d021893e : 00000235`234bfac0 00000000`001f0003 00000000`00000000 00000235`234bfac0 : KERNELBASE!WaitForMultipleObjectsEx+0xef
02 00007ffd`c9be987e : 00000000`00000000 00000000`00000000 00000000`00000000 00007ffd`c9be8637 : KERNELBASE!WaitForMultipleObjects+0xe
03 00007ffd`c9be9cfa : 00000000`00000002 000000a3`0bfff880 000000a3`0bfff868 00000000`000003e8 : TmUmEvt64!TmmonDestoryAddonObject+0x6b6e
04 00000000`6dfef046 : 00000000`00000000 00000000`00000124 00000000`00000000 00000000`00000000 : TmUmEvt64!TmmonDestoryAddonObject+0x6fea
05 00000000`6df97a87 : 00000235`2344fcd0 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0xdf046
06 00000000`6df97c2e : 00000000`6e021ec0 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87a87
07 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x87c2e
08 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
09 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

#  3  Id: 770.1540 Suspend: 1 Teb: 000000a3`0ba6d000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ff6`32a4d5dc : 00000235`234e7650 00000000`00000000 cccccccc`cccccccc cccccccc`00000000 : KERNELBASE!DebugBreak+0x2
01 00007ff6`32a4d78e : 00000000`000001cc 00000235`234e5750 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!log_read+0x8c [D:\Devel\Repos\fluent-bit\src\flb_log.c @ 102] 
02 00007ff6`32a4c387 : 00000235`234e5750 00000235`234e4320 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!log_worker_collector+0x14e [D:\Devel\Repos\fluent-bit\src\flb_log.c @ 131] 
03 00007ff6`32cdd924 : 00000235`234e4320 00000000`00000000 cccccccc`cccccccc cccccccc`cccccccc : fluent_bit!step_callback+0x77 [D:\Devel\Repos\fluent-bit\src\flb_worker.c @ 54] 
04 00007ff6`32d48bb0 : 00000235`234ef500 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!pthread_create_wrapper+0x74 [D:\Devel\Repos\fluent-bit\lib\monkey\mk_core\external\winpthreads.c @ 654] 
05 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : fluent_bit!thread_start<unsigned int (__cdecl*)(void *),1>+0x50 [minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 97] 
06 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
07 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   4  Id: 770.9b4 Suspend: 1 Teb: 000000a3`0ba6f000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`cedcfb19 : 00000000`00000000 00000000`00000000 00007ffd`cedc0000 00000000`00000000 : ntdll!NtRemoveIoCompletion+0x14
01 00007ffd`d07a84d4 : 00007ffd`cedc2c70 00000235`234d8c50 00000000`00000000 00000000`00000000 : mswsock!SockAsyncThread+0x79
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   5  Id: 770.aec Suspend: 1 Teb: 000000a3`0ba73000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d30c9eae : 00000235`234d7898 00000235`234d7898 00000235`234bde98 00000000`00000001 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x76e
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

   6  Id: 770.1a54 Suspend: 1 Teb: 000000a3`0ba75000 Unfrozen
 # RetAddr           : Args to Child                                                           : Call Site
00 00007ffd`d30c9eae : 00000235`234d7898 00000235`234d7898 00000235`234bde98 00000000`00000001 : ntdll!NtWaitForWorkViaWorkerFactory+0x14
01 00007ffd`d07a84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x76e
02 00007ffd`d310e8b1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
03 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
0:003> dx Debugger.Sessions[0].Processes[1904].Threads[5440].Stack.Frames[1].SwitchTo();dv /t /v
Debugger.Sessions[0].Processes[1904].Threads[5440].Stack.Frames[1].SwitchTo()
000000a3`0c0ffbb0 int64 fd = 0n460
000000a3`0c0ffbb8 struct flb_log * log = 0x00000235`234e5750
000000a3`0c0feb60 int bytes = 0n1176
000000a3`0c0feb80 struct log_message msg = struct log_message
0:003> dx -r1 (*((fluent_bit!log_message *)0xa30c0feb80))
(*((fluent_bit!log_message *)0xa30c0feb80))                 [Type: log_message]
    [+0x000] size             : 0x303230322d687361 [Type: unsigned __int64]
    [+0x008] msg              : ".06.17","_type":"_doc","_id":"KxpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49539,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"LBpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49490,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"LRpBwHIBnLAzIfGdSKMn","_version":1,"result":"creat... [Type: char [4088]]
0:003> db 000000a3`0c0feb80 L0n1176
000000a3`0c0feb80  61 73 68 2d 32 30 32 30-2e 30 36 2e 31 37 22 2c  ash-2020.06.17",
000000a3`0c0feb90  22 5f 74 79 70 65 22 3a-22 5f 64 6f 63 22 2c 22  "_type":"_doc","
000000a3`0c0feba0  5f 69 64 22 3a 22 4b 78-70 42 77 48 49 42 6e 4c  _id":"KxpBwHIBnL
000000a3`0c0febb0  41 7a 49 66 47 64 53 4b-4d 6e 22 2c 22 5f 76 65  AzIfGdSKMn","_ve
000000a3`0c0febc0  72 73 69 6f 6e 22 3a 31-2c 22 72 65 73 75 6c 74  rsion":1,"result
000000a3`0c0febd0  22 3a 22 63 72 65 61 74-65 64 22 2c 22 5f 73 68  ":"created","_sh
000000a3`0c0febe0  61 72 64 73 22 3a 7b 22-74 6f 74 61 6c 22 3a 32  ards":{"total":2
000000a3`0c0febf0  2c 22 73 75 63 63 65 73-73 66 75 6c 22 3a 32 2c  ,"successful":2,
000000a3`0c0fec00  22 66 61 69 6c 65 64 22-3a 30 7d 2c 22 5f 73 65  "failed":0},"_se
000000a3`0c0fec10  71 5f 6e 6f 22 3a 34 39-35 33 39 2c 22 5f 70 72  q_no":49539,"_pr
000000a3`0c0fec20  69 6d 61 72 79 5f 74 65-72 6d 22 3a 31 2c 22 73  imary_term":1,"s
000000a3`0c0fec30  74 61 74 75 73 22 3a 32-30 31 7d 7d 2c 7b 22 69  tatus":201}},{"i
000000a3`0c0fec40  6e 64 65 78 22 3a 7b 22-5f 69 6e 64 65 78 22 3a  ndex":{"_index":
000000a3`0c0fec50  22 6c 6f 67 73 74 61 73-68 2d 32 30 32 30 2e 30  "logstash-2020.0
000000a3`0c0fec60  36 2e 31 37 22 2c 22 5f-74 79 70 65 22 3a 22 5f  6.17","_type":"_
000000a3`0c0fec70  64 6f 63 22 2c 22 5f 69-64 22 3a 22 4c 42 70 42  doc","_id":"LBpB
000000a3`0c0fec80  77 48 49 42 6e 4c 41 7a-49 66 47 64 53 4b 4d 6e  wHIBnLAzIfGdSKMn
000000a3`0c0fec90  22 2c 22 5f 76 65 72 73-69 6f 6e 22 3a 31 2c 22  ","_version":1,"
000000a3`0c0feca0  72 65 73 75 6c 74 22 3a-22 63 72 65 61 74 65 64  result":"created
000000a3`0c0fecb0  22 2c 22 5f 73 68 61 72-64 73 22 3a 7b 22 74 6f  ","_shards":{"to
000000a3`0c0fecc0  74 61 6c 22 3a 32 2c 22-73 75 63 63 65 73 73 66  tal":2,"successf
000000a3`0c0fecd0  75 6c 22 3a 32 2c 22 66-61 69 6c 65 64 22 3a 30  ul":2,"failed":0
000000a3`0c0fece0  7d 2c 22 5f 73 65 71 5f-6e 6f 22 3a 34 39 34 39  },"_seq_no":4949
000000a3`0c0fecf0  30 2c 22 5f 70 72 69 6d-61 72 79 5f 74 65 72 6d  0,"_primary_term
000000a3`0c0fed00  22 3a 31 2c 22 73 74 61-74 75 73 22 3a 32 30 31  ":1,"status":201
000000a3`0c0fed10  7d 7d 2c 7b 22 69 6e 64-65 78 22 3a 7b 22 5f 69  }},{"index":{"_i
000000a3`0c0fed20  6e 64 65 78 22 3a 22 6c-6f 67 73 74 61 73 68 2d  ndex":"logstash-
000000a3`0c0fed30  32 30 32 30 2e 30 36 2e-31 37 22 2c 22 5f 74 79  2020.06.17","_ty
000000a3`0c0fed40  70 65 22 3a 22 5f 64 6f-63 22 2c 22 5f 69 64 22  pe":"_doc","_id"
000000a3`0c0fed50  3a 22 4c 52 70 42 77 48-49 42 6e 4c 41 7a 49 66  :"LRpBwHIBnLAzIf
000000a3`0c0fed60  47 64 53 4b 4d 6e 22 2c-22 5f 76 65 72 73 69 6f  GdSKMn","_versio
000000a3`0c0fed70  6e 22 3a 31 2c 22 72 65-73 75 6c 74 22 3a 22 63  n":1,"result":"c
000000a3`0c0fed80  72 65 61 74 65 64 22 2c-22 5f 73 68 61 72 64 73  reated","_shards
000000a3`0c0fed90  22 3a 7b 22 74 6f 74 61-6c 22 3a 32 2c 22 73 75  ":{"total":2,"su
000000a3`0c0feda0  63 63 65 73 73 66 75 6c-22 3a 32 2c 22 66 61 69  ccessful":2,"fai
000000a3`0c0fedb0  6c 65 64 22 3a 30 7d 2c-22 5f 73 65 71 5f 6e 6f  led":0},"_seq_no
000000a3`0c0fedc0  22 3a 34 39 34 39 31 2c-22 5f 70 72 69 6d 61 72  ":49491,"_primar
000000a3`0c0fedd0  79 5f 74 65 72 6d 22 3a-31 2c 22 73 74 61 74 75  y_term":1,"statu
000000a3`0c0fede0  73 22 3a 32 30 31 7d 7d-5d 7d 0a 00 00 00 00 00  s":201}}]}......
000000a3`0c0fedf0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee00  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee10  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee20  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee30  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee40  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee50  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee60  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee70  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee80  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fee90  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feea0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feeb0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feec0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feed0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feee0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feef0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef00  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef10  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef20  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef30  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef40  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef50  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef60  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef70  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef80  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fef90  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fefa0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fefb0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fefc0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fefd0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0fefe0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0feff0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0ff000  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000000a3`0c0ff010  00 00 00 00 00 00 00 00

Then I continued execution at the perror code.

Looking at standard error that was being logged to file by winsw, I see the following:

[2020/06/17 03:10:03] [debug] [output:es:es.0] Elasticsearch response
{"took":14,"errors":false,"items":[{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"HhpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49536,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"HxpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49487,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"IBpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49488,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"IRpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49489,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"IhpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49488,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"IxpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49539,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"JBpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49537,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"JRpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49526,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"JhpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49527,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"JxpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49538,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"KBpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49489,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"KRpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49490,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"KhpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49540,"_primary_term":1,"status":201}},{"index":{"_index":"logstÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌÌflb_pipe_r msg.size: No error

The garbage near the end combines perfectly with the buffer text from the debug break:

ash-2020.06.17","_type":"_doc","_id":"KxpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49539,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"LBpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49490,"_primary_term":1,"status":201}},{"index":{"_index":"logstash-2020.06.17","_type":"_doc","_id":"LRpBwHIBnLAzIfGdSKMn","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":49491,"_primary_term":1,"status":201}}]}

This is interesting as it also indicates that the first buffer was probably not null terminated.

This then points to a problem with the strlen in flb_log_print, which is technically redundant since vsnprintf returns the number of characters written (excluding the terminating null), and this points to a possible problem with the va_args to flb_log_print where one is probably a string that is not null terminated.

gitfool on 17 Jun 2020

Going to add more debugging to see if i can get more info and/or catch these anomalies upstream.

gitfool on 17 Jun 2020

@gitfool Thank you for the effort. That's great evidence.

This is interesting as it also indicates that the first buffer was probably not null terminated.

Hmm. I guess that the problem is not the message per se, but the
flb_pipe_r() reading it in a partial manner.

What do you think about the fix f92ebf7e? I'm guessing that this can
solve your issue.

fujimotos on 17 Jun 2020

Yeah I suspect write_all/read_all will be needed for this to work as expected. Also, it could be smarter and read the size then read size bytes, rather than always reading the full buffer (4KB), but that complicates it compared to always writing/reading the same fixed size buffer, and 4KB is probably small enough to not worry about.

gitfool on 17 Jun 2020

@gitfool So I updated win32-2251 with a few relevant fixes.
If our investigation so far are correct, this should resolve
the crash issue.

Commit: 2dc44840
AppVeyor: builds/33567497

I'd appreciate if you can check this out and see if it actually works.

fujimotos on 17 Jun 2020

@fujimotos okay, I've dropped in the artifacts from AppVeyor and am running a test over night.

gitfool on 17 Jun 2020

@fujimotos I added some debugging to satisfy my curiosity and find out exactly how these changes behaved.

flb_pipe_write_all _always_ sent all the data in one hit. No drama here.

flb_pipe_read_all _usually_ received all the data in one hit, but at seemingly random times it would receive partial data. Interestingly, in such cases the first read was always a multiple of the TCP MSS:

MSS = MTU - HDR = 1500 - 40 = 1460 bytes

I saw the following combinations of reads that fit 4KB:

1460 + 2636 = 4096
2920 + 1176 = 4096

This matches nicely with my first crash analysis above where I later commented:

Also, when it does fail, bytes seems to always be 2636.

I was also worried about the following code:
https://github.com/fluent/fluent-bit/blob/2dc44840e3fcf6103962e22c94801d263cdc3a56/src/flb_log.c#L400-L412
... but have concluded it works as expected due to the local initializer that zeroes the struct combined with passing a smaller buffer to vsnprintf.

TL;DR: your changes LGTM! 🎉

gitfool on 18 Jun 2020

@gitfool Great! Now I'll prepare to push these fixes to mainline.
So thankful for your investigation and testing!

fujimotos on 19 Jun 2020

👍1

Close this issue for #2295 being merged

fujimotos on 26 Jun 2020

Was this page helpful?

0 / 5 - 0 ratings