Logstash: OutOfMemoryError after upgrading Logstash

[joshuaspence]

We just recently updated Logstash from version 2.3.4 to 2.4.0 and the Logstash service stops running after 5-10 minutes with the following error showing up in /var/log/logstash/logstash.err:

Error: Your application used more memory than the safety cap of 1G.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

Our configuration was working fine with Logstash 2.3.4 and hasn't changed since updating, which makes me think that this could be a Logstash bug? I'm not too familiar with Java, so I'm not sure what additional information I can provide here, but let me know if you require any additional details.

[jsvd]

if you're able to attach the configuration and a heap dump it would be great :) I can pm you a link to place the heapdump if you don't want to share it publicly.

[joshuaspence]

Here is our configuration:

input {
  beats {
    port => 5044
  }
}
input {
  file {
    add_field => {
      "role" => "logstash_shipper"
    }
    path => "/var/log/logstash/logstash.log"
    type => "logstash"
  }
}
input {
  heartbeat {
    add_field => {
      "role" => "logstash_shipper"
    }
    interval => "10"
    type => "heartbeat"
  }
}
filter {
  metrics {
    add_field => {
      type => "metric"
    }
    add_tag => "metric"
    meter => ["events"]
  }
}
output {
  if [type] == "heartbeat" {
    http {
      http_method => "get"
      url => "http://127.0.0.1:8500/v1/agent/check/pass/service:logstash:1"
    }
  }
}
output {
  sqs {
    batch => false
    queue => "logstash"
    region => "us-east-1"
    workers => 16
  }
}
output {
  if "metric" in [tags] {
    statsd {
      count => {
        "logstash_events" => "%{[events][count]}"
      }
      gauge => {
        "logstash_event_rate.1m" => "%{[events][rate_1m]}"
        "logstash_event_rate.5m" => "%{[events][rate_5m]}"
        "logstash_event_rate.15m" => "%{[events][rate_15m]}"
      }
    }
  }
}

Yeah, I would prefer not to share the heap dump publicly so if you could PM me a link that'd be great.

[jsvd]

@joshuaspence link sent to your email.

[ph]

@joshuaspence The heap will help us to see if any bytebuf aren't correctly freed,.

[ph]

We took a look at the heap dump, its seems that we have a memory leak around the org.jruby.ext.openssl.x509store, The only output that uses Ruby's SSL implementation in your config is SQS.

@jsvd did some investigation on his side, this bug seems to be introduced with the upgrade to jruby 1.7.25 that ships with jruby-openssl 0.9.16

[joshuaspence]

Just noticed that this issue still has the "need details" label. Is there anything else that you require from me?

[jsvd]

@joshuaspence thanks for providing all this information. I've stopped debugging this for a week now..unfortunately I haven't been able to replicate this situation yet.
Next week I'll give it another go. My main suspicion still falls on the openssl that ships with jruby-1.7.25, which was only recently included in logstash.

[joshuaspence]

I don't know if it is the same issue, but we just had an OutOfMemoryError again on Logstash 2.4.1, using the following plugins:

logstash-codec-collectd (2.0.4)
logstash-codec-dots (2.0.4)
logstash-codec-edn (2.0.4)
logstash-codec-edn_lines (2.0.4)
logstash-codec-es_bulk (2.0.4)
logstash-codec-fluent (2.0.4)
logstash-codec-graphite (2.0.4)
logstash-codec-json (2.1.4)
logstash-codec-json_lines (2.1.3)
logstash-codec-line (2.1.2)
logstash-codec-msgpack (2.0.4)
logstash-codec-multiline (2.0.11)
logstash-codec-netflow (2.1.1)
logstash-codec-oldlogstashjson (2.0.4)
logstash-codec-plain (2.0.4)
logstash-codec-rubydebug (2.0.7)
logstash-filter-anonymize (2.0.4)
logstash-filter-checksum (2.0.4)
logstash-filter-clone (2.0.6)
logstash-filter-csv (2.1.3)
logstash-filter-date (2.1.6)
logstash-filter-dns (2.1.3)
logstash-filter-drop (2.0.4)
logstash-filter-fingerprint (2.0.5)
logstash-filter-geoip (2.0.7)
logstash-filter-grok (2.0.5)
logstash-filter-json (2.0.6)
logstash-filter-kv (2.1.0)
logstash-filter-metrics (3.0.2)
logstash-filter-multiline (2.0.5)
logstash-filter-mutate (2.0.6)
logstash-filter-ruby (2.0.5)
logstash-filter-sleep (2.0.4)
logstash-filter-split (2.0.5)
logstash-filter-syslog_pri (2.0.4)
logstash-filter-throttle (2.0.4)
logstash-filter-urldecode (2.0.4)
logstash-filter-useragent (2.0.8)
logstash-filter-uuid (2.0.5)
logstash-filter-xml (2.2.0)
logstash-input-beats (3.1.7)
logstash-input-couchdb_changes (2.0.4)
logstash-input-elasticsearch (2.0.5)
logstash-input-eventlog (3.0.3)
logstash-input-exec (2.0.6)
logstash-input-file (2.2.5)
logstash-input-ganglia (2.0.6)
logstash-input-gelf (2.0.7)
logstash-input-generator (2.0.4)
logstash-input-graphite (2.0.7)
logstash-input-heartbeat (2.0.4)
logstash-input-http (2.2.3)
logstash-input-http_poller (2.1.0)
logstash-input-imap (2.0.5)
logstash-input-irc (2.0.5)
logstash-input-jdbc (3.1.0)
logstash-input-kafka (2.0.9)
logstash-input-log4j (2.0.7)
logstash-input-lumberjack (2.0.7)
logstash-input-pipe (2.0.4)
logstash-input-rabbitmq (4.1.0)
logstash-input-redis (2.0.6)
logstash-input-s3 (2.0.6)
logstash-input-snmptrap (2.0.4)
logstash-input-sqs (2.0.5)
logstash-input-stdin (2.0.4)
logstash-input-syslog (2.0.5)
logstash-input-tcp (3.0.6)
logstash-input-twitter (2.2.2)
logstash-input-udp (2.0.5)
logstash-input-unix (2.0.6)
logstash-input-xmpp (2.0.5)
logstash-input-zeromq (2.0.4)
logstash-output-cloudwatch (2.0.4)
logstash-output-csv (2.0.5)
logstash-output-elasticsearch (2.7.1)
logstash-output-email (3.0.5)
logstash-output-exec (2.0.5)
logstash-output-file (2.2.5)
logstash-output-ganglia (2.0.4)
logstash-output-gelf (2.0.5)
logstash-output-graphite (2.0.5)
logstash-output-hipchat (3.0.4)
logstash-output-http (2.1.3)
logstash-output-irc (2.0.4)
logstash-output-juggernaut (2.0.4)
logstash-output-kafka (2.0.5)
logstash-output-lumberjack (2.0.6)
logstash-output-nagios (2.0.4)
logstash-output-nagios_nsca (2.0.5)
logstash-output-null (2.0.4)
logstash-output-opentsdb (2.0.4)
logstash-output-pagerduty (2.0.4)
logstash-output-pipe (2.0.4)
logstash-output-rabbitmq (3.1.2)
logstash-output-redis (2.0.5)
logstash-output-s3 (2.0.7)
logstash-output-sns (3.0.4)
logstash-output-sqs (2.0.5)
logstash-output-statsd (2.0.7)
logstash-output-stdout (2.0.6)
logstash-output-tcp (2.0.4)
logstash-output-udp (2.0.4)
logstash-output-xmpp (2.0.4)
logstash-output-zeromq (2.1.0)
logstash-patterns-core (2.0.5)

I can provide another heapdump if it would help. We want to update to Logstash 5 (which might fix this issue for us), but are currently blocking on awslabs/logstash-output-amazon_es#42 and logstash-plugins/logstash-filter-prune#17.

[joshuaspence]

Is there anything I can do to help with this issue? Our logging rig is dying due to OOM every 1-2 days at the moment.

[ph]

@joshuaspence I know you are now using 5.x it is still an issue?
@jsvd Any other luck or update on this issue?

[joshuaspence]

I'm not observing this problem anymore. We have changed a bunch of things in our Logstash setup since filing this ticket, one of which likely fixed whatever the issue was.