Fluentd: Using monitor_agent with workers 8 causes Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"

Created on 26 Jan 2018 · 4Comments · Source: fluent/fluentd

Having workers above 4 with monitor_agent enabled causes Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224" errors on some workers. Here is our aggregation config.

<source>
  @type forward
  port 24224
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    hosts xxx:9200,xxx:9200,xxx:9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    flush_interval 20s
    flush_at_shutdown true
    buffer_type file
    buffer_path /var/log/td-agent/buffer/elasticsearch
    buffer_chunk_limit 512m
    buffer_queue_limit 10
    request_timeout 120s
    reload_on_failure true
    resurrect_after 30
    num_threads 8
  </store>
  <store>
    @type s3
    s3_bucket xxx
    s3_region xxx
    s3_object_key_format  %{time_slice}/events/events_%{index}.%{file_extension}
    path logs/
    buffer_type file
    buffer_path /var/log/td-agent/buffer/s3
    buffer_chunk_limit 512m
    buffer_queue_limit 10
    flush_at_shutdown true
    time_slice_format %Y-%m-%d/%H
    time_slice_wait 5m
    utc
    store_as lzma2
    num_threads 16
  </store>
</match>
<system>
  workers 8
</system>
<source>
  @type monitor_agent
  port 24220
</source>

2018-01-26 18:12:59 +0000 [info]: #4 starting fluentd worker pid=96716 ppid=95834 worker=4
2018-01-26 18:12:59 +0000 [info]: #4 listening port port=24224 bind="0.0.0.0"
2018-01-26 18:12:59 +0000 [error]: #4 unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/socket.rb:201:in `bind'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/socket.rb:201:in `listen'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/socket.rb:750:in `block in tcp_server_sockets'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/socket.rb:227:in `each'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/socket.rb:227:in `foreach'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/socket.rb:748:in `tcp_server_sockets'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/utils.rb:65:in `create_listeners'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:128:in `listen'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:109:in `initialize'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/httpserver.rb:47:in `initialize'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_monitor_agent.rb:242:in `new'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/plugin/in_monitor_agent.rb:242:in `start'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:165:in `block in start'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:153:in `each'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:153:in `block in lifecycle'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:140:in `each'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:140:in `lifecycle'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/root_agent.rb:164:in `start'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/engine.rb:274:in `start'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/engine.rb:219:in `run'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:774:in `run_engine'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:523:in `block in run_worker'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:699:in `main_process'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/supervisor.rb:518:in `run_worker'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.1.0/bin/fluentd:8:in `<top (required)>'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/bin/fluentd:22:in `load'
  2018-01-26 18:12:59 +0000 [error]: #4 /opt/td-agent/embedded/bin/fluentd:22:in `<main>'
2018-01-26 18:12:59 +0000 [error]: #4 unexpected error error_class=Errno::EADDRINUSE error="Address already in use - bind(2) for 0.0.0.0:24224"
  2018-01-26 18:12:59 +0000 [error]: #4 suppressed same stacktrace
2018-01-26 18:12:59 +0000 [info]: Worker 4 finished unexpectedly with status 1

If you remove

<source>
  @type monitor_agent
  port 24220
</source>

or you drop workers to 4:

<system>
  workers 4
</system>

the error goes away and td-agent starts as expected. Here are the versions installed:

fluent-logger (0.7.1)
fluent-plugin-elasticsearch (2.4.1)
fluent-plugin-kafka (0.6.5)
fluent-plugin-rewrite-tag-filter (2.0.1)
fluent-plugin-s3 (1.1.0)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-systemd (0.3.1)
fluent-plugin-td (1.0.0)
fluent-plugin-td-monitoring (0.2.3)
fluent-plugin-webhdfs (1.2.2)
fluentd (1.0.2)

This happens with td-agent-3.1.1-0.el7.x86_64 on Fedora 26. Upgrading the fluentd gem to 1.1.0 does not fix the issue.

Source