Logstash: Beats input: the pipeline is blocked, temporary refusing new connection.

Created on 17 Dec 2015 · 8Comments · Source: elastic/logstash

Logstash and Filbeat doesn't work together.

Logstash

$ /opt/logstash/bin/logstash -V
logstash 2.1.1

/var/log/logstash/logstash.log

{:timestamp=>"2015-12-17T14:31:22.817000-0800", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
{:timestamp=>"2015-12-17T14:29:54.160000-0800", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-12-17T14:29:54.163000-0800", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::CircuitBreaker::HalfOpenBreaker, :level=>:warn}

Filebeat

$ filebeat -version
filebeat version 1.0.0 (amd64)

/var/log/filebeat/filebeat.log

    2015-12-17T13:16:45-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:41035->192.168.0.139:5044: i/o timeout
    2015-12-17T13:19:42-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:46896->192.168.0.136:5044: i/o timeout
    2015-12-17T13:19:42-08:00 ERR Error sending/writing event: read tcp 192.168.2.231:46896->192.168.0.136:5044: i/o timeout
    2015-12-17T13:22:55-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:58461->192.168.0.59:5044: i/o timeout
    2015-12-17T13:34:15-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:59216->192.168.0.59:5044: i/o timeout
    2015-12-17T13:41:44-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:59733->192.168.0.59:5044: i/o timeout
    2015-12-17T13:42:34-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:59794->192.168.0.59:5044: i/o timeout
    2015-12-17T13:42:34-08:00 ERR Error sending/writing event: read tcp 192.168.2.231:59794->192.168.0.59:5044: i/o timeout
    2015-12-17T13:43:26-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:48502->192.168.0.136:5044: i/o timeout
    2015-12-17T13:51:48-08:00 ERR SSL client failed to connect with: read tcp 192.168.2.231:49066->192.168.0.136:5044: i/o timeout

Source

alexanderilyin

Most helpful comment

I set my congestion threshold to 40. I remember needing to tune this to get a value that worked.

congestion_threshold => "40"

dhdanno on 7 Jun 2016

👍3

All 8 comments

Please raise this against the beats input, here - https://github.com/logstash-plugins/logstash-input-beats/issues

Providing your configs would also be useful :)

markwalkom on 21 Dec 2015

@alexanderilyin this typically happens when components downstream (outputs in LS) cannot handle the load you are sending. Also as Mark mentioned, providing your configs will help. We can help you debug here: https://discuss.elastic.co/c/logstash. Github is mainly used for issues/enhancements. Thanks!

suyograo on 21 Dec 2015

@markharwood @suyograo is it possible that such problems could be caused by multiline filter? I mean in my configs last event will always be "in memory" / "in processing" until new multiline message will appear in log file and if there a lot of hosts with suck files logstash could run out of some internal resources?

I mean my case looks like that it. After some uptime client side (logstash-forwarder or filebeats could not deliver event because of i/o erros). Restarting of client side does not help. When I restart logstash everything start working again.

Are there some "workers" limits which could be increase in logstash?

filter {
  if [type] == 'php-fpm-slow-log' {
     # Drop empty lines
     if [message] =~ /^\s*$/ {
         drop { }
     }

     # Time prefix appears on each line
     multiline {
         pattern => "^(\[0x0|script_filename|$)"
         what    => "previous"
     }

     grok {
         patterns_dir => "/etc/logstash/patterns/"
         match => [ "message", "(?m)%{PHP_FPM_SLOW_LOG}" ]
     }

     date {
         match        => [ "timestamp" , "dd-MMM-yyyy HH:mm:ss" ]
         target       => "@timestamp"
         remove_field => "timestamp"
     }
  }
}

h3. UPDATE

I've disabled all multiline filters and problem is gone.
I have 3 ES servers in cluster and each has LS on it.
filebeats delivers event s from 15 hosts.
At present time there are ~21000 messages per our added to ES.
Disabled fpm logs generated not more that 100 events per hour.
My output section looks like:

output {
#  stdout {
#    codec => rubydebug
#  }
  elasticsearch {
    hosts => [
        "1.example.com:9200",
        "2.example.com:9200",
        "3.example.com:9200"
    ]
#    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
#    document_type => "%{[@metadata][type]}"
  }
}

alexanderilyin on 22 Dec 2015

Increase the congestion_threshold value on the beats input. Fixed it for me.

dhdanno on 10 Mar 2016

Hi Guys.,

we are getting below error in logstash log

log:

Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover
Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn

Note : Logstash instance was working fine for moderate load but when load increased drastically we are facing this issue and logstash is unable to come out of it. To avoid this
we have increased logstash congestion_threshold to 25 also increased -w to 10 for 4 core machine for logstash service still it is in same state.