Fluentd: Fluentd 0.14.11 deadlocked in file output
td-agent 3/fluentd 0.14.11, running on CentOS 6, writing to a file. Senders were all buffering. Multiple sigdumps were identical other than the allocation count, so I feel fairly confident things were deadlocked.
I had a hard time killing the daemon when it was in this state
2017-04-18 21:44:20 +0000 [warn]: thread doesn't exit correctly (killed or other reason) plugin=Fluent::Plugin::FileOutput title=:flush_thread_0 thread=#<Thread:0x007f092a15c720@/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil
We rolled back to td-agent 2/fluentd 0.12, and may have seen the same thing, but I am not sure, and we can no longer reproduce the problem.
Sigdump at 2017-04-18 20:47:33 +0000 process 3303 (/usr/sbin/td-agent)
Thread #<Thread:0x007f0931faf988> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:179:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:179:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:725:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:478:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:650:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:473:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/command/fluentd.rb:288:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007f092b55d940> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f092b55d508> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f092a15c720> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:900:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1013:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1234:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f092a15c450> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1197:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f092741f898> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f092741f398> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f092741eec0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f0927413ef8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_monitor_agent.rb:234:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f0927413638> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f0927df3ec8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:848:in `block in handle_stream_with_custom_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:783:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:847:in `handle_stream_with_custom_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:775:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:710:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/event_router.rb:90:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:292:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:203:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:240:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:239:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:239:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:248:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/server.rb:465:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
Thread #<Thread:0x007f0927df16c8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:149:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:149:in `log_event_loop'
GC stat:
count: 3840
heap_allocated_pages: 273
heap_sorted_length: 273
heap_allocatable_pages: 0
heap_available_slots: 111273
heap_live_slots: 110894
heap_free_slots: 379
heap_final_slots: 0
heap_marked_slots: 66052
heap_eden_pages: 273
heap_tomb_pages: 0
total_allocated_pages: 273
total_freed_pages: 0
total_allocated_objects: 142062923
total_freed_objects: 141952029
malloc_increase_bytes: 35536
malloc_increase_bytes_limit: 29739514
minor_gc_count: 3648
major_gc_count: 192
remembered_wb_unprotected_objects: 1658
remembered_wb_unprotected_objects_limit: 2376
old_objects: 63003
old_objects_limit: 121542
oldmalloc_increase_bytes: 35984
oldmalloc_increase_bytes_limit: 92131169
Built-in objects:
111,273: TOTAL
27,789: T_STRING
22,271: T_DATA
20,514: T_ARRAY
18,339: T_IMEMO
6,865: T_RATIONAL
6,812: T_BIGNUM
2,492: T_CLASS
1,848: T_OBJECT
1,241: T_NODE
1,114: T_HASH
590: T_REGEXP
277: FREE
277: T_ICLASS
271: T_FILE
258: T_STRUCT
227: T_MODULE
73: T_SYMBOL
9: T_FLOAT
5: T_MATCH
1: T_COMPLEX
All objects:
28,024: String
11,516: Time
9,453: Array
3,711: Rational
3,658: Integer
1,431: Class
993: Hash
665: Proc
600: Gem::Requirement
590: Regexp
430: Gem::Dependency
227: Module
217: File
208: Thread::Mutex
180: Fluent::Plugin::Buffer::Metadata
177: Gem::Version
106: Fluent::Plugin::Buffer::FileChunk
101: Encoding
92: Coolio::IO::Watcher
77: Gem::Specification
77: Gem::StubSpecification::StubLine
77: Gem::StubSpecification
75: Fluent::Config::ConfigureProxy
73: Symbol
56: Range
46: IO::Buffer
45: Fluent::PluginHelper::Server::TCPCallbackSocket
45: Fluent::PluginHelper::Server::EventHandler::TCPServer
45: TCPSocket
42: MessagePack::Buffer
42: MessagePack::Unpacker
27: MatchData
24: Fluent::Config::Section
17: Fluent::Config::Element
16: OptionParser::Switch::RequiredArgument
13: Thread::Backtrace
13: Fluent::Plugin::Base::State
12: Thread
12: OptionParser::Switch::NoArgument
11: Method
9: Float
8: Fluent::Registry
6: OptionParser::OptionMap
5: IO
4: Monitor
4: Object
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: Coolio::Loop
3: Fluent::Plugin::Output::FlushThreadState
3: Fluent::EventRouter::Rule
3: Fluent::TimeFormatter
3: Fluent::Plugin::OutFileFormatter
3: Fluent::Plugin::FileBuffer
3: Fluent::Plugin::FileOutput
3: Fluent::PluginHelper::Server::ServerInfo
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Zlib::GzipWriter
2: Coolio::TCPServer
2: ThreadGroup
2: Fluent::GlobMatchPattern
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: ThreadSafe::Cache
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: UnboundMethod
1: WEBrick::HTTPServer::MountTable
1: Fluent::Plugin::SyslogInput
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: Fluent::Plugin::ForwardInput
1: Fluent::MessagePackEventStream
1: EOFError
1: Strptime
1: Fluent::TimeParser
1: Fluent::Plugin::SyslogParser
1: Fluent::AllMatchPattern
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::Agent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: Fluent::EngineClass
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Compat::NullOutputChain
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: WEBrick::HTTPVersion
1: Set
1: OptionParser
1: Fluent::Plugin::MonitorAgentInput
1: IRB::Notifier::NoMsgNotifier
1: #<Class:0x007f092b4a7d98>
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: Gem::PathSupport
1: #<Class:0x007f092b51e560>
1: Gem::Platform
1: fatal
1: Data
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Fluent::Log
1: Logger::Formatter
1: ServerEngine::DaemonLogger
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser::CompletingHash
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 19,199,333 bytes
Array 1 elements
Hash 14 pairs
hfinucane
All 41 comments
Thanks for the report.
I first heard this out_file issue. Is this reproducible?
And does v0.14.15 also have same problem?
repeatedly
on 25 Apr 2017
It was reproduceable in my environment for about a day, but I can no longer reproduce after rolling back and then forward again. I was hoping the sigdump would have enough information to debug.
hfinucane
on 26 Apr 2017
I caught something similar running td-agent 0.14.16-
Sigdump at 2017-06-07 20:02:50 +0000 process 1160 (/usr/sbin/td-agent)
Thread #<Thread:0x007f54d4db60c0> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:224:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:224:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:737:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:503:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:661:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:498:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007f54cb4a33d8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cb4a2fc8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4f4ae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4efdb8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4ef688> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4ee210> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4e6ba0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4e5278> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4de018> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4dce98> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cd1be4e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4d7808> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4d5be8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4d4518> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4cfdd8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cc4cf770> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2675c0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca267250> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca266f08> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca266b70> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2666e8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca266260> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca265dd8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2659c8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca265338> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca264aa0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca264780> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca264398> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca267b60> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca257968> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2575a8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca257210> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca256e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca256a40> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca256658> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca255898> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca255528> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca255258> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca255028> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca254dd0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca254b50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2548d0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca254678> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2543a8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ca2540d8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d5bf58> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d5b5a8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d5ad88> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d5a400> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d59a50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d592a8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d58678> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d6fbc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d6e090> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d6ca88> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d6c2e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d7e238> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d7d450> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d7c7f8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d7c050> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d8e660> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d9e268> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d9c0a8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4dad088> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4dbfa30> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4dcfbd8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4dccd70> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4dcc3c0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4dde160> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cebafe30> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54cebae170> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54ceacdad0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:879:in `block in handle_stream_with_custom_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:878:in `handle_stream_with_custom_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:797:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/event_router.rb:159:in `emit_events'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007f54d4d641a8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:192:in `sleep'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:192:in `log_event_loop'
GC stat:
count: 511419
heap_allocated_pages: 721
heap_sorted_length: 809
heap_allocatable_pages: 0
heap_available_slots: 293869
heap_live_slots: 292972
heap_free_slots: 897
heap_final_slots: 0
heap_marked_slots: 197865
heap_eden_pages: 721
heap_tomb_pages: 0
total_allocated_pages: 1793
total_freed_pages: 1072
total_allocated_objects: 52329380397
total_freed_objects: 52329087425
malloc_increase_bytes: 139456
malloc_increase_bytes_limit: 16777216
minor_gc_count: 500707
major_gc_count: 10712
remembered_wb_unprotected_objects: 786
remembered_wb_unprotected_objects_limit: 1572
old_objects: 195575
old_objects_limit: 341804
oldmalloc_increase_bytes: 414784
oldmalloc_increase_bytes_limit: 19088575
Built-in objects:
293,869: TOTAL
131,344: T_STRING
44,559: T_DATA
36,110: T_ARRAY
25,132: T_HASH
17,995: T_IMEMO
14,368: T_RATIONAL
14,315: T_BIGNUM
3,619: T_OBJECT
2,421: T_CLASS
1,392: T_NODE
821: FREE
578: T_REGEXP
554: T_STRUCT
283: T_ICLASS
230: T_MODULE
96: T_SYMBOL
38: T_FILE
9: T_FLOAT
4: T_MATCH
1: T_COMPLEX
All objects:
130,658: String
24,558: Hash
21,772: Time
20,407: Array
7,214: Rational
7,160: Integer
1,382: Class
1,024: Fluent::EventRouter::Pipeline::FilterOptimizer
1,024: Fluent::EventRouter::Pipeline
590: Proc
578: Regexp
533: Gem::Requirement
420: Fluent::Plugin::Buffer::Metadata
371: Gem::Dependency
230: Module
178: Gem::Version
131: Thread::Mutex
101: Encoding
96: Symbol
80: Thread::Backtrace
74: Thread
73: Gem::Specification
73: Gem::StubSpecification::StubLine
73: Gem::StubSpecification
72: Fluent::Config::ConfigureProxy
65: Fluent::Plugin::Output::FlushThreadState
48: Range
28: Fluent::Config::Section
26: MatchData
22: Coolio::IO::Watcher
21: Fluent::Config::Element
17: Method
17: OptionParser::Switch::RequiredArgument
15: File
15: Fluent::Plugin::Base::State
13: OptionParser::Switch::NoArgument
11: IO::Buffer
10: Fluent::PluginHelper::Server::TCPCallbackSocket
10: Fluent::PluginHelper::Server::EventHandler::TCPServer
10: TCPSocket
9: Float
8: Fluent::Registry
6: OptionParser::OptionMap
5: Fluent::Plugin::Buffer::FileChunk
5: EOFError
5: Fluent::EventRouter::Rule
5: IO
4: Monitor
4: Fluent::GlobMatchPattern
4: Object
3: Coolio::Loop
3: Fluent::Plugin::FileOutput
3: Fluent::Plugin::FileBuffer
3: OptionParser::List
3: Fluent::TimeFormatter
3: Fluent::Plugin::OutFileFormatter
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: Fluent::PluginHelper::Server::ServerInfo
3: TCPServer
2: Fluent::EventTime
2: MessagePack::Buffer
2: MessagePack::Unpacker
2: Fluent::Plugin::IllumioUnwrapFilter
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: OptionParser::Switch::PlacedArgument
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
2: Coolio::TCPServer
2: ThreadGroup
2: Fluent::PluginHelper::Server::UDPCallbackSocket
1: Fluent::MultiEventStream
1: Fluent::MessagePackEventStream
1: Fluent::Plugin::SyslogInput
1: Fluent::Compat::NullOutputChain
1: Strptime
1: Fluent::TimeParser
1: Fluent::Plugin::SyslogParser
1: ArgumentError
1: Fluent::Plugin::MonitorAgentInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EngineClass
1: Set
1: Fluent::Log
1: Logger::Formatter
1: ServerEngine::DaemonLogger
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: Fluent::AllMatchPattern
1: Fluent::Plugin::ForwardInput
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Gem::PathSupport
1: Gem::Platform
1: #<Class:0x007f54ce2d04b8>
1: #<Class:0x007f54ce30d778>
1: fatal
1: Data
1: WEBrick::HTTPServer::MountTable
1: OptionParser::CompletingHash
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
1: WEBrick::Log
1: Thread::SizedQueue
1: WEBrick::HTTPServer
String 4,243,574 bytes
Array 4 elements
Hash 1 pairs
There are a ton more threads, and I'm not sure if it's the same or not, but the symptoms are the same- deadlocked daemon with no files getting written.
hfinucane
on 7 Jun 2017
I am also experiencing what appears to be the same issue. RHEL 6, fluentd gem 0.14.16. sigdump from the deadlocked worker is below, followed by the sigdump from a functional worker (note the lack of locks).
Edit: Moved to attachments instead of giant code blocks.
scottbuettner
on 14 Aug 2017
Here is another threaddump, from after all 4 worker processes deadlocked.
threaddump.txt
I have just updated to fluentd 0.14.20, I'll let you know if the issue persists.
scottbuettner
on 15 Aug 2017
0.14.20 has had one of our 4 worker processes deadlock already, here is it's threaddump. I expect the others to deadlock shortly.
threaddump_14.20.txt
scottbuettner
on 15 Aug 2017
Thanks for the testing. Could you paste your configuration and traffic info?
I want to reproduce the issue on my environment.
repeatedly
on 16 Aug 2017
Greetings we are seeing what appears to be deadlocks like these. We're processing a few hundred million events a day on 4 servers. What can I do to help with diagnostics? These are all 14.20 and a couple of the servers (though they have identical gems and versions) run well, while others have issues.
GLStephen
on 24 Aug 2017
I'm waiting on some corporate approval on how much information I can share. What I can say right now is that our config looks generally like this, and it's operating on messages originally received from syslog.
<match some.tag>
@type file
append true
format custom_format
path /some/path/${host}/%Y-%m-%d/${tag[3]}_%H
<buffer tag,host,time>
path "/other/path/buffer/"
flush_mode interval
flush_interval 60s
flush_thread_count 2
chunk_limit_size 64MB
total_limit_size 50GB
timekey 1h
timekey_wait 300
</buffer>
</match>
Thanks for the info! I will try to reproduce it on my environment.
repeatedly
on 29 Aug 2017
Hmm... I use the modified version of scottbuettner's conf, but I still can't reproduce the problem on my Mac and Ubuntu 16.04.
Here is my test set:
- conf
<source>
@type forward
</source>
<source>
@type forward
port 24225
</source>
# To increase lock / unlock, add test field.
<match test.tag>
@type file
append true
format json
path /home/repeatedly/tmp/deadlock/log/${host}/${test}/%Y-%m-%d/${tag[1]}_%H
<buffer tag,host,test,time>
path /home/repeatedly/tmp/deadlock/buffer/
flush_mode interval
flush_interval 60s
flush_thread_count 2
chunk_limit_size 32MB
total_limit_size 50GB
timekey 1h
timekey_wait 300
flush_at_shutdown true
</buffer>
</match>
- client
require 'fluent-logger'
require 'json'
port = ARGV[0].nil? ? 24224 : Integer(ARGV[0])
log = Fluent::Logger::FluentLogger.new(nil, :host=>'localhost', :port => port)
msgs = []
# hoge.json contains the number of nginx logs. Different content is ok.
File.read('hoge.json').each_line { |l|
m = JSON.parse(l)
m['host'] = "host#{rand(50)}"
m['test'] = "test#{rand(10)}"
msgs << m
}
i = 0
size = msgs.size
loop {
(0..100000).each { |i|
msg = msgs[i % size]
msg['num'] = i
log.post("test.tag", msg)
sleep 0.0001
}
sleep 1
}
I run 2 or 3 clients in parallel but the problem is not reproduced in 12 hours.
Does anyone provide reproducible step for this problem locally?
repeatedly
on 6 Sep 2017
@repeatedly Can you track how many messages you're actually sending with that? We're receiving somewhere around 3000/second in the system that fails, and it usually does so within a few hours.
I don't think it's relevant, but we're running on RHEL/CentOS 6, rather than Ubuntu. td-agent bundles the ruby interpreter so I don't think it would be much difference there, but I thought I'd mention it.
scottbuettner
on 6 Sep 2017
Reporter here, also running CentOS 6.
hfinucane
on 7 Sep 2017
I setup CentOS 6 box and restarted reproduce step with 3000msgs/second.
repeatedly
on 15 Sep 2017
Hmm... I can't still reproduce this problem on CentOS 6.
I use ruby 2.1.10 or ruby 2.4.1 with fluentd v0.14.21 and run fluentd / logger in 5 days.
Does anyone provide the more detailed information, e.g. hardward spec?
And if you hit the problem again, could you try perf or pid2line.rb? https://docs.fluentd.org/v0.12/articles/trouble-shooting#high-cpu-usage-issue
In addition, ruby 2.4 has better deadlock detection.
So does anyone try above configuration with ruby 2.4 on reproducible node?
I want to know my setup/configuration can reproduce the problem or not.
repeatedly
on 19 Sep 2017
t2.large in AWS- it was never doing much throughput, cpu usage was pretty minimal, even during bursts.
hfinucane
on 20 Sep 2017
More deadlock info:
Sigdump at 2017-10-03 15:03:53 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 60
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 168837
heap_free_slots: 1936
heap_final_slots: 0
heap_marked_slots: 104339
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1696085
total_freed_objects: 1527248
malloc_increase_bytes: 2865440
malloc_increase_bytes_limit: 16777216
minor_gc_count: 52
major_gc_count: 8
remembered_wb_unprotected_objects: 15144
remembered_wb_unprotected_objects_limit: 20242
old_objects: 85273
old_objects_limit: 156152
oldmalloc_increase_bytes: 4218592
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
69,874: T_STRING
24,039: T_ARRAY
23,536: T_IMEMO
16,273: T_DATA
11,641: T_HASH
7,488: T_FILE
5,336: T_STRUCT
4,957: T_OBJECT
2,525: T_CLASS
1,860: FREE
1,392: T_NODE
576: T_REGEXP
505: T_BIGNUM
320: T_ICLASS
235: T_MODULE
72: T_MATCH
69: T_SYMBOL
65: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
61,302: String
15,786: Array
9,523: Hash
9,341: Time
5,898: File
4,749: Fluent::Plugin::Buffer::Metadata
3,014: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
2,376: Proc
1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
1,463: Class
677: Gem::Requirement
576: Regexp
478: Gem::Dependency
307: MessagePack::Buffer
292: Integer
287: MessagePack::Packer
260: Range
235: Module
214: Gem::Version
101: Encoding
94: MatchData
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
73: Fluent::MultiEventStream
69: Symbol
60: Rational
57: Fluent::Config::ConfigureProxy
44: Coolio::IO::Watcher
27: Thread::Backtrace
22: IO::Buffer
22: TCPSocket
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
20: MessagePack::Unpacker
20: File::Stat
19: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
9: Float
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
7: ThreadError
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: EOFError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MessagePackEventStream
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 4,271,107 bytes
Array 1 elements
Hash 223 pairs
Sigdump at 2017-10-03 15:04:13 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 60
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 157455
heap_free_slots: 13318
heap_final_slots: 0
heap_marked_slots: 104339
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1698768
total_freed_objects: 1541313
malloc_increase_bytes: 538912
malloc_increase_bytes_limit: 16777216
minor_gc_count: 52
major_gc_count: 8
remembered_wb_unprotected_objects: 15144
remembered_wb_unprotected_objects_limit: 20242
old_objects: 85273
old_objects_limit: 156152
oldmalloc_increase_bytes: 1892064
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
62,795: T_STRING
23,579: T_IMEMO
22,634: T_ARRAY
15,655: T_DATA
13,242: FREE
9,860: T_HASH
7,483: T_FILE
5,028: T_STRUCT
4,900: T_OBJECT
2,525: T_CLASS
1,392: T_NODE
576: T_REGEXP
320: T_ICLASS
292: T_BIGNUM
235: T_MODULE
118: T_MATCH
69: T_SYMBOL
60: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
62,969: String
16,141: Array
9,527: Hash
9,343: Time
5,899: File
4,749: Fluent::Plugin::Buffer::Metadata
3,015: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
2,377: Proc
1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
1,463: Class
677: Gem::Requirement
576: Regexp
478: Gem::Dependency
307: MessagePack::Buffer
292: Integer
287: MessagePack::Packer
260: Range
235: Module
214: Gem::Version
142: MatchData
101: Encoding
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
73: Fluent::MultiEventStream
69: Symbol
60: Rational
57: Fluent::Config::ConfigureProxy
44: Coolio::IO::Watcher
44: Thread::Backtrace
22: IO::Buffer
22: TCPSocket
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
20: MessagePack::Unpacker
20: File::Stat
19: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
9: Float
8: ThreadError
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: EOFError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MessagePackEventStream
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 4,322,053 bytes
Array 2 elements
Hash 223 pairs
Sigdump at 2017-10-03 15:05:17 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 60
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 159739
heap_free_slots: 11034
heap_final_slots: 0
heap_marked_slots: 104339
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1701414
total_freed_objects: 1541675
malloc_increase_bytes: 622512
malloc_increase_bytes_limit: 16777216
minor_gc_count: 52
major_gc_count: 8
remembered_wb_unprotected_objects: 15144
remembered_wb_unprotected_objects_limit: 20242
old_objects: 85273
old_objects_limit: 156152
oldmalloc_increase_bytes: 1975664
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
64,449: T_STRING
23,742: T_IMEMO
23,030: T_ARRAY
15,674: T_DATA
10,958: FREE
9,863: T_HASH
7,484: T_FILE
5,028: T_STRUCT
4,900: T_OBJECT
2,525: T_CLASS
1,392: T_NODE
576: T_REGEXP
320: T_ICLASS
292: T_BIGNUM
235: T_MODULE
166: T_MATCH
69: T_SYMBOL
60: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
64,623: String
16,535: Array
9,530: Hash
9,344: Time
5,900: File
4,749: Fluent::Plugin::Buffer::Metadata
3,016: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
2,378: Proc
1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
1,463: Class
677: Gem::Requirement
576: Regexp
478: Gem::Dependency
307: MessagePack::Buffer
292: Integer
287: MessagePack::Packer
260: Range
235: Module
214: Gem::Version
190: MatchData
101: Encoding
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
73: Fluent::MultiEventStream
69: Symbol
60: Thread::Backtrace
60: Rational
57: Fluent::Config::ConfigureProxy
44: Coolio::IO::Watcher
22: IO::Buffer
22: TCPSocket
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
20: MessagePack::Unpacker
20: File::Stat
19: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
9: Float
8: ThreadError
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: EOFError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MessagePackEventStream
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 4,372,518 bytes
Array 2 elements
Hash 223 pairs
Sigdump at 2017-10-03 15:05:21 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 60
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 161963
heap_free_slots: 8810
heap_final_slots: 0
heap_marked_slots: 104339
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1704000
total_freed_objects: 1542037
malloc_increase_bytes: 706112
malloc_increase_bytes_limit: 16777216
minor_gc_count: 52
major_gc_count: 8
remembered_wb_unprotected_objects: 15144
remembered_wb_unprotected_objects_limit: 20242
old_objects: 85273
old_objects_limit: 156152
oldmalloc_increase_bytes: 2059264
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
66,103: T_STRING
23,905: T_IMEMO
23,366: T_ARRAY
15,693: T_DATA
9,866: T_HASH
8,734: FREE
7,485: T_FILE
5,028: T_STRUCT
4,900: T_OBJECT
2,525: T_CLASS
1,392: T_NODE
576: T_REGEXP
320: T_ICLASS
292: T_BIGNUM
235: T_MODULE
214: T_MATCH
69: T_SYMBOL
60: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
66,277: String
16,869: Array
9,533: Hash
9,345: Time
5,901: File
4,749: Fluent::Plugin::Buffer::Metadata
3,017: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
2,379: Proc
1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
1,463: Class
677: Gem::Requirement
576: Regexp
478: Gem::Dependency
307: MessagePack::Buffer
292: Integer
287: MessagePack::Packer
260: Range
238: MatchData
235: Module
214: Gem::Version
101: Encoding
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
76: Thread::Backtrace
73: Fluent::MultiEventStream
69: Symbol
60: Rational
57: Fluent::Config::ConfigureProxy
44: Coolio::IO::Watcher
22: IO::Buffer
22: TCPSocket
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
20: MessagePack::Unpacker
20: File::Stat
19: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
9: Float
8: ThreadError
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: EOFError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MessagePackEventStream
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 4,422,977 bytes
Array 2 elements
Hash 223 pairs
Sigdump at 2017-10-03 15:06:01 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 60
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 164243
heap_free_slots: 6530
heap_final_slots: 0
heap_marked_slots: 104339
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1706643
total_freed_objects: 1542400
malloc_increase_bytes: 791136
malloc_increase_bytes_limit: 16777216
minor_gc_count: 52
major_gc_count: 8
remembered_wb_unprotected_objects: 15144
remembered_wb_unprotected_objects_limit: 20242
old_objects: 85273
old_objects_limit: 156152
oldmalloc_increase_bytes: 2144288
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
67,768: T_STRING
24,068: T_IMEMO
23,743: T_ARRAY
15,714: T_DATA
9,870: T_HASH
7,486: T_FILE
6,454: FREE
5,028: T_STRUCT
4,901: T_OBJECT
2,525: T_CLASS
1,392: T_NODE
576: T_REGEXP
320: T_ICLASS
292: T_BIGNUM
262: T_MATCH
235: T_MODULE
69: T_SYMBOL
60: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
67,942: String
17,244: Array
9,537: Hash
9,347: Time
5,902: File
4,749: Fluent::Plugin::Buffer::Metadata
3,018: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
2,380: Proc
1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
1,463: Class
677: Gem::Requirement
576: Regexp
478: Gem::Dependency
307: MessagePack::Buffer
292: Integer
287: MessagePack::Packer
286: MatchData
260: Range
235: Module
214: Gem::Version
101: Encoding
93: Thread::Backtrace
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
73: Fluent::MultiEventStream
69: Symbol
60: Rational
57: Fluent::Config::ConfigureProxy
44: Coolio::IO::Watcher
22: IO::Buffer
22: TCPSocket
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
20: MessagePack::Unpacker
20: File::Stat
19: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
9: ThreadError
9: Float
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: EOFError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MessagePackEventStream
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 4,473,868 bytes
Array 2 elements
Hash 223 pairs
Sigdump at 2017-10-03 15:06:23 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 60
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 166485
heap_free_slots: 4288
heap_final_slots: 0
heap_marked_slots: 104339
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1709247
total_freed_objects: 1542762
malloc_increase_bytes: 874736
malloc_increase_bytes_limit: 16777216
minor_gc_count: 52
major_gc_count: 8
remembered_wb_unprotected_objects: 15144
remembered_wb_unprotected_objects_limit: 20242
old_objects: 85273
old_objects_limit: 156152
oldmalloc_increase_bytes: 2227888
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
69,422: T_STRING
24,231: T_IMEMO
24,097: T_ARRAY
15,733: T_DATA
9,873: T_HASH
7,487: T_FILE
5,028: T_STRUCT
4,901: T_OBJECT
4,212: FREE
2,525: T_CLASS
1,392: T_NODE
576: T_REGEXP
320: T_ICLASS
310: T_MATCH
292: T_BIGNUM
235: T_MODULE
69: T_SYMBOL
60: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
69,596: String
17,596: Array
9,540: Hash
9,348: Time
5,903: File
4,749: Fluent::Plugin::Buffer::Metadata
3,019: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
2,381: Proc
1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
1,463: Class
677: Gem::Requirement
576: Regexp
478: Gem::Dependency
334: MatchData
307: MessagePack::Buffer
292: Integer
287: MessagePack::Packer
260: Range
235: Module
214: Gem::Version
109: Thread::Backtrace
101: Encoding
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
73: Fluent::MultiEventStream
69: Symbol
60: Rational
57: Fluent::Config::ConfigureProxy
44: Coolio::IO::Watcher
22: IO::Buffer
22: TCPSocket
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
20: MessagePack::Unpacker
20: File::Stat
19: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
9: ThreadError
9: Float
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: EOFError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MessagePackEventStream
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 4,524,327 bytes
Array 2 elements
Hash 223 pairs
Sigdump at 2017-10-03 15:07:54 +0000 process 887 (/usr/sbin/td-agent)
Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
/opt/td-agent/embedded/bin/fluentd:22:in `load'
/opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
/usr/sbin/td-agent:7:in `load'
/usr/sbin/td-agent:7:in `<main>'
Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
/opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
/opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
GC stat:
count: 61
heap_allocated_pages: 419
heap_sorted_length: 419
heap_allocatable_pages: 0
heap_available_slots: 170773
heap_live_slots: 167229
heap_free_slots: 3544
heap_final_slots: 0
heap_marked_slots: 107398
heap_eden_pages: 419
heap_tomb_pages: 0
total_allocated_pages: 419
total_freed_pages: 0
total_allocated_objects: 1711942
total_freed_objects: 1544713
malloc_increase_bytes: 58032
malloc_increase_bytes_limit: 16777216
minor_gc_count: 53
major_gc_count: 8
remembered_wb_unprotected_objects: 15506
remembered_wb_unprotected_objects_limit: 20242
old_objects: 86119
old_objects_limit: 156152
oldmalloc_increase_bytes: 2268288
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
170,773: TOTAL
69,478: T_STRING
24,394: T_IMEMO
24,292: T_ARRAY
15,754: T_DATA
9,781: T_HASH
7,488: T_FILE
5,001: T_STRUCT
4,881: T_OBJECT
3,867: FREE
2,525: T_CLASS
1,392: T_NODE
576: T_REGEXP
358: T_MATCH
320: T_ICLASS
292: T_BIGNUM
235: T_MODULE
69: T_SYMBOL
60: T_RATIONAL
9: T_FLOAT
1: T_COMPLEX
All objects:
47,724: String
6,013: Time
5,900: File
4,647: Array
4,595: Fluent::Plugin::Buffer::Metadata
3,016: Thread::Mutex
2,947: Fluent::Plugin::Buffer::FileChunk
1,463: Class
923: Hash
677: Gem::Requirement
576: Regexp
540: Proc
478: Gem::Dependency
235: Module
214: Gem::Version
101: Encoding
91: Gem::Specification
91: Gem::StubSpecification::StubLine
91: Gem::StubSpecification
69: Symbol
59: Rational
57: Fluent::Config::ConfigureProxy
46: Range
44: Coolio::IO::Watcher
27: MatchData
22: IO::Buffer
22: Thread::Backtrace
21: Fluent::PluginHelper::Server::TCPCallbackSocket
21: Fluent::PluginHelper::Server::EventHandler::TCPServer
21: TCPSocket
18: MessagePack::Buffer
18: MessagePack::Unpacker
17: Thread
17: OptionParser::Switch::RequiredArgument
15: Fluent::Config::Section
13: Fluent::Config::Element
13: OptionParser::Switch::NoArgument
12: File::Stat
9: Float
8: Fluent::Plugin::Output::FlushThreadState
8: Fluent::Registry
7: Method
7: IO
7: Integer
6: Fluent::Plugin::Base::State
6: OptionParser::OptionMap
4: Object
3: ThreadError
3: TCPServer
3: Fluent::PluginHelper::EventLoop::DefaultWatcher
3: ThreadGroup
3: Monitor
3: Coolio::Loop
3: OptionParser::List
3: IRB::Notifier::LeveledNotifier
3: JSON::Ext::Generator::State
2: Fluent::MultiEventStream
2: EOFError
2: Fluent::PluginHelper::Server::ServerInfo
2: Fluent::Log
2: Logger::Formatter
2: ServerEngine::DaemonLogger
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: MessagePack::Factory
2: OpenSSL::PKey::DH
2: UnboundMethod
1: Fluent::MessagePackEventStream
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: Coolio::TCPServer
1: WEBrick::HTTPServer::MountTable
1: Thread::SizedQueue
1: WEBrick::HTTPServer
1: WEBrick::Log
1: DRb::DRbMessage
1: DRb::DRbTCPSocket
1: DRb::DRbServer
1: Fluent::PluginHelper::Timer::TimerWatcher
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::Plugin::ForwardOutput::Node
1: Fluent::Plugin::ForwardOutput::FailureDetector
1: Fluent::Plugin::SecondaryFileOutput
1: ArgumentError
1: DRb::DRbIdConv
1: Fluent::EngineClass
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::Plugin::MonitorAgentInput
1: StringScanner
1: Fluent::LogDeviceIO
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: WEBrick::HTTPVersion
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Fluent::PluginLogger
1: Fluent::Plugin::DebugAgentInput
1: Fluent::Plugin::ForwardOutput
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: #<Class:0x007fb4ac4daf18>
1: Gem::PathSupport
1: #<Class:0x007fb4ac536b38>
1: fatal
1: Data
1: Fluent::Plugin::FileBuffer
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: Complex
1: IOError
1: Binding
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
String 3,842,341 bytes
Array 1 elements
Hash 223 pairs
note @repeatedly the deadlock posted above happens with flush_thread_count > 1 (I had it set to 8) almost immediately. With flush_thread_count set to 1 the deadlock certainly does not happen immediately, but I am waiting to see if it happens over a longer period of time. Hopefully that helps.
GLStephen
on 3 Oct 2017
With threads at 1, it took about 2 hours to fail.
GLStephen
on 4 Oct 2017
pid2line.rb result from slack
$ /opt/td-agent/embedded/bin/ruby pid2line.rb 887
ruby state:S utime:99 LWPID:887 EIP:7fb4b456ea82 -> ba82
Elf vaddr: ba82 -> ba82 in /usr/lib64/libpthread-2.17.so
__pthread_cond_timedwait
:?
ruby-timer-thr state:S utime:0 LWPID:888 EIP:7fb4b3b51dfd -> ecdfd
Elf vaddr: ecdfd -> ecdfd in /usr/lib64/libc-2.17.so
__poll_nocancel
:?
thread.rb:70 state:S utime:18 LWPID:991 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:17 LWPID:992 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:19 LWPID:993 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:18 LWPID:994 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:16 LWPID:995 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:18 LWPID:996 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:18 LWPID:997 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:16 LWPID:998 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:6 LWPID:999 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70 state:S utime:60 LWPID:1000 EIP:7fb4b3b5cd13 -> f7d13
Elf vaddr: f7d13 -> f7d13 in /usr/lib64/libc-2.17.so
__epoll_wait_nocancel
:?
drb.rb:1498 state:S utime:0 LWPID:1001 EIP:7fb4b3b53ba3 -> eeba3
Elf vaddr: eeba3 -> eeba3 in /usr/lib64/libc-2.17.so
__select_nocancel
:?
thread.rb:70 state:S utime:6 LWPID:1002 EIP:7fb4b3b5cd13 -> f7d13
Elf vaddr: f7d13 -> f7d13 in /usr/lib64/libc-2.17.so
__epoll_wait_nocancel
:?
thread.rb:70 state:S utime:4 LWPID:1003 EIP:7fb4b3b53ba3 -> eeba3
Elf vaddr: eeba3 -> eeba3 in /usr/lib64/libc-2.17.so
__select_nocancel
:?
thread.rb:70 state:S utime:351 LWPID:1004 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
@repeatedly I've been working through this deadlock issue and I had a question about the retry process, is that a separate thread? or does it consume the thread that is active? How is retry stored? memory? is it reasonable that a huge number of chunks in a retry state could lead to what we are seeing here, or at least in my case? I was looking at some libraries and behavior and it seems that some of the plugins were stuck in a retry loop because of a library version that was upgraded from the gemspec that has an issue, the question is whether that matters, if that only consumes 1 thread and just keeps retrying the 1 chunk until it's tossed then it won't cause an increasing resource usage, if retries are piled up though the behavior would be different
GLStephen
on 10 Oct 2017
I had a question about the retry process, is that a separate thread? or does it consume the thread that is active?
Does "retry process" mean buffer chunk retry, right?
If so, retry is done by same thread for flush. Output plugin shares retry state and if the flush of one thread is succeeded, retry state is cleared.
How is retry stored? memory?
Yes. See the code: https://github.com/fluent/fluentd/blob/629c470a2a8412dd51875f8adbea0ea7b47dc914/lib/fluent/plugin/output.rb#L1134
it seems that some of the plugins were stuck in a retry loop because of a library version that was upgraded from the gemspec that has an issue,
Does this 'stuck' is different from this issue? Is it known issue between plugin/library versions?
it won't cause an increasing resource usage,
Yeah. retry itself doesn't much increase resource usage.
Does this deadlock happen after some errors?
repeatedly
on 11 Oct 2017
I tried to reproduce this issue in 629c470a2a8412dd51875f8adbea0ea7b47dc914 by the following steps and I found that deadlock occurs if an input plugin thread locks chunks in Buffer#write by chunk.mon_enter and an enqueue thread locks the buffer in Buffer#enqueue_chunk at the same time.
1. Add the following change
diff --git a/lib/fluent/plugin/buffer.rb b/lib/fluent/plugin/buffer.rb
index be66f3e8..d4c6be87 100644
--- a/lib/fluent/plugin/buffer.rb
+++ b/lib/fluent/plugin/buffer.rb
@@ -496,6 +496,7 @@ module Fluent
if stored
block.call(chunk, adding_bytesize)
+ sleep 3
end
end
2. Prepare a configuration file
<source>
@type forward
</source>
<match test.*>
@type file
format json
path /tmp/fluentd/file/${tag}-%Y%m%d
<buffer time, tag>
path /tmp/fluentd/buffer/test.*.buffer
flush_mode interval
flush_interval 1s
timekey 1d
</buffer>
</match>
3. Launch fluentd
ruby bin/fluentd -c deadlock.conf
4. Post events
require 'fluent-logger'
logger = Fluent::Logger::FluentLogger.new(nil, host: 'localhost', port: 24224)
logger.send(:write, ['test.a', [[Time.now.to_i - 86400, { msg: 0 }], [Time.now.to_i, { msg: 1 }]]])
5. sigdump
Sigdump at 2017-10-11 17:24:33 +0900 process 84647 (bin/fluentd)
Thread #<Thread:0x007fb08107ef70> status=run priority=0
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/engine.rb:228:in `sleep'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/engine.rb:228:in `run'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:774:in `run_engine'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:523:in `block in run_worker'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:699:in `main_process'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:518:in `run_worker'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
bin/fluentd:5:in `<main>'
Thread #<Thread:0x007fb080c6e6b0> status=sleep priority=0
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:304:in `queued?'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:1015:in `next_flush_time'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:1289:in `flush_thread_run'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:438:in `block (2 levels) in start'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb080c6e430> status=sleep priority=0
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:353:in `each'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:1249:in `enqueue_thread_run'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
Thread #<Thread:0x007fb080c6e228> status=sleep priority=0
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `lock'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:457:in `write_once'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:207:in `block in write'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:206:in `each'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:206:in `write'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:897:in `block in handle_stream_with_custom_format'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:823:in `write_guard'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:896:in `handle_stream_with_custom_format'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:815:in `execute_chunking'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:750:in `emit_buffered'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/event_router.rb:96:in `emit_stream'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:320:in `on_message'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/io.rb:123:in `on_readable'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/io.rb:186:in `on_readable'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/loop.rb:88:in `run_once'
/Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/loop.rb:88:in `run'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
/Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
GC stat:
count: 54
heap_allocated_pages: 377
heap_sorted_length: 409
heap_allocatable_pages: 32
heap_available_slots: 153663
heap_live_slots: 153127
heap_free_slots: 536
heap_final_slots: 0
heap_marked_slots: 100124
heap_eden_pages: 377
heap_tomb_pages: 0
total_allocated_pages: 377
total_freed_pages: 0
total_allocated_objects: 1135049
total_freed_objects: 981922
malloc_increase_bytes: 5486912
malloc_increase_bytes_limit: 16777216
minor_gc_count: 47
major_gc_count: 7
remembered_wb_unprotected_objects: 1162
remembered_wb_unprotected_objects_limit: 2320
old_objects: 97483
old_objects_limit: 194974
oldmalloc_increase_bytes: 5710432
oldmalloc_increase_bytes_limit: 16777216
Built-in objects:
153,663: TOTAL
76,920: T_STRING
34,229: T_ARRAY
17,853: T_IMEMO
11,396: T_OBJECT
5,465: T_NODE
2,324: T_CLASS
2,125: T_HASH
1,580: T_DATA
543: T_REGEXP
460: FREE
255: T_ICLASS
232: T_MODULE
71: T_RATIONAL
66: T_STRUCT
58: T_SYMBOL
28: T_MATCH
26: T_BIGNUM
22: T_FILE
9: T_FLOAT
1: T_COMPLEX
All objects:
69,320: String
25,693: Array
4,931: Gem::Requirement
3,377: Gem::Dependency
1,899: Hash
1,318: Class
844: Time
769: Gem::StubSpecification
769: Gem::StubSpecification::StubLine
769: Gem::Specification
596: Gem::Version
543: Regexp
382: Proc
232: Module
101: Encoding
76: RubyVM::InstructionSequence
71: Rational
69: Fluent::Config::ConfigureProxy
58: Symbol
53: Range
46: MatchData
30: Thread::Mutex
28: Binding
26: Integer
17: OptionParser::Switch::RequiredArgument
16: File
14: Thread::Backtrace
13: OptionParser::Switch::NoArgument
10: Fluent::Config::Section
9: Float
8: Method
8: Fluent::Config::Element
8: Fluent::Registry
6: OptionParser::OptionMap
4: Thread
4: Coolio::IO::Watcher
4: Fluent::Plugin::Buffer::Metadata
4: Fluent::Plugin::Base::State
4: Object
3: IO
3: OptionParser::List
3: JSON::Ext::Generator::State
3: IRB::Notifier::LeveledNotifier
3: UnboundMethod
3: MessagePack::Buffer
2: Fluent::PluginHelper::Server::EventHandler::TCPServer
2: OptionParser::Switch::PlacedArgument
2: OptionParser::Switch::OptionalArgument
2: ThreadSafe::Cache
2: BigDecimal
2: Fluent::PluginHelper::Server::ServerInfo
2: IO::Buffer
2: OpenSSL::PKey::DH
2: Monitor
2: MessagePack::Unpacker
2: Fluent::Plugin::Output::PlaceholderValidator
2: MessagePack::Factory
2: Fluent::Plugin::Buffer::FileChunk
1: TCPSocket
1: StringScanner
1: Fluent::Log
1: Logger::Formatter
1: ServerEngine::DaemonLogger
1: WEBrick::HTTPVersion
1: #<Class:0x007fb08082e208>
1: #<Class:0x007fb0808547c8>
1: IRB::Notifier::CompositeNotifier
1: IRB::StdioOutputMethod
1: IRB::Notifier::NoMsgNotifier
1: Errno::EEXIST
1: Fluent::Compat::NullOutputChain
1: Fluent::Plugin::ForwardInput
1: LoadError
1: Fluent::Plugin::JSONFormatter
1: ArgumentError
1: Coolio::Loop
1: TCPServer
1: Fluent::PluginHelper::EventLoop::DefaultWatcher
1: Fluent::Plugin::Output::FlushThreadState
1: Complex
1: ThreadGroup
1: IOError
1: RubyVM
1: NoMemoryError
1: SystemStackError
1: Random
1: ARGF.class
1: fatal
1: Data
1: Gem::PathSupport
1: Set
1: OptionParser::CompletingHash
1: Gem::Platform
1: Fluent::EngineClass
1: URI::RFC2396_Parser
1: URI::RFC3986_Parser
1: OpenSSL::X509::Store
1: Resolv::DNS::Config
1: Resolv::DNS
1: Resolv::Hosts
1: Resolv
1: Fluent::EventRouter::MatchCache
1: Fluent::EventRouter
1: Fluent::NoMatchMatch
1: Fluent::RootAgent
1: Fluent::SystemConfig
1: Fluent::Supervisor::LoggerInitializer
1: Fluent::Supervisor
1: OptionParser
1: MessagePack::Packer
1: Fluent::PluginHelper::Server::EventHandler::UDPServer
1: UDPSocket
1: IPAddr
1: Coolio::TCPServer
1: Fluent::Plugin::FileOutput
1: Fluent::Plugin::FileBuffer
1: Fluent::GlobMatchPattern
1: Fluent::EventRouter::Rule
1: Fluent::EventTime
1: Fluent::MultiEventStream
1: Fluent::PluginHelper::Server::TCPCallbackSocket
String 4,318,695 bytes
Array 3 elements
Hash 0 pairs
abicky
on 11 Oct 2017
👍2
@abicky Thanks. I will check it!
repeatedly
on 11 Oct 2017
It is reproduced on my env.
The problem is no buffer lock in write. It causes lock order mismatch.
For the performance, giant lock in write is not good.
I'm now trying to fix the problem.
repeatedly
on 12 Oct 2017
👍3
@repeatedly I'm not 100% up on fluentd internals so just trying to be helpful, but is it possible to create a system like used for timekey and timekey_wait, but in reverse. Stop writing to a buffer chunk before it is enqueud, then when enqueue occurs the lock is simply unnecessary. This could operate by monitoring "flush_percentage" or something similar. If the lock is problematic, try to avoid needing one.
GLStephen
on 13 Oct 2017
@repeatedly Thank you for your effort :)
Do you have any update on this issue?
If you are busy to fix the problem, I'll also try to fix it.
abicky
on 17 Oct 2017
@abicky If you have a patch for this, it is very helpful. Because we don't have actual production environment for this problem, so if your patch resolve your production issue, this is reliable for other users :)
repeatedly
on 17 Oct 2017
@repeatedly Okay, I'll try to write a patch 馃挭
abicky
on 17 Oct 2017
👍3
@abicky Currently, I am facing the same issue. I hope this patch, too !
mkishi
on 17 Oct 2017
@repeatedly @abicky We are seeing this issue happening all the time (v14.22.rc1). Our archive servers deadlock in about 5 minutes from restart. Using append and compress may make this happen more easily(?).
Here's our out_file configuration:
<match **>
@type file
path /data/log/${tag}/${tag}
<buffer tag,time>
@type file
path /data/buffer
flush_mode interval
flush_interval 10
flush_at_shutdown true
total_limit_size 1G
timekey_use_utc true
</buffer>
<format>
@type json
</format>
include_time_key true
append true
compress gzip
</match>
mpeltonen
on 18 Oct 2017
I've created https://github.com/fluent/fluentd/pull/1721 to resolve this issue.
abicky
on 20 Oct 2017
Does anyone test above patch on your traffic / setting?
repeatedly
on 20 Oct 2017
As I commented in https://github.com/fluent/fluentd/pull/1721#issuecomment-338371066, I've checked it worked fine in our development environment although the number of events is less than the number in the production environment.
I'll try it in our production environment after the release of a new RC version or your review. In our production environment deadlock occurs almost everyday, so it takes a few days to test it in a more complicated environment.
Of course, I checked the patch resolved the problem in the case of https://github.com/fluent/fluentd/issues/1549#issuecomment-335736547.
abicky
on 21 Oct 2017
I've just applied the patched gem and have started testing it. Usually my environment was deadlocking in 1-3 hours, I should be able to report back tomorrow with the results.
scottbuettner
on 22 Oct 2017
so far so good! 22 hours in and no deadlocks, running with 4 worker processes. @repeatedly any ideas when this can hit an RC or actual release version?
Big thank you to @abicky for the patch!
scottbuettner
on 23 Oct 2017
@scottbuettner any notable impact on performance? I'm very encouraged that this patch will solve the issues we've been seeing also!
GLStephen
on 23 Oct 2017
@GLStephen I don't have any benchmarks to compare against or anything that detailed, but I didn't notice a drop in performance from just comparing CPU/load/etc.
scottbuettner
on 23 Oct 2017
I released v0.14.22.rc2 for testing.
repeatedly
on 24 Oct 2017
🎉1
Looks promising for us too! No deadlocks seen so far with .22.rc2. Thanks a lot @abicky for the patch!
mpeltonen
on 24 Oct 2017
thanks for the patch @bicky , @repeatedly this seems to fix our issue as well, we'll fully load it over the weekend and know for sure, but we are not seeing the problematic behaviors we were seeing before at all
GLStephen
on 27 Oct 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
dmnfortytwo
路
3Comments
amread
路
5Comments
aniamin
路
7Comments
agrozyme
路
3Comments
robert-7
路
4Comments
Most helpful comment
@repeatedly Okay, I'll try to write a patch 馃挭