Logstash: Logstash syslog input causing a "UDP Listener died" error

[nullbuddy1243]

Hi all,

I am trying to receive syslog input from a syslog server which is being fed logs from Docker containers via logspout.
I am rather sure that I configured the input correctly to receive syslog on port 5000. Here is my input config for syslog

syslog {
    port => 5000
    type => "docker_log"
}

And here is my full input config on pastebin: http://pastebin.com/qcX5tFTi

Here is the error I am receiving: http://pastebin.com/jJxZsULX

    {:timestamp=>"2015-05-09T21:17:44.676000+0000", :message=>"UDP listener died", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-0.1.3/lib/logstash/inputs/udp.rb:79:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-0.1.3/lib/logstash/inputs/udp.rb:49:in `run'", "/opt/logstash/lib/logstash/pipeline.rb:174:in `inputworker'", "/opt/logstash/lib/logstash/pipeline.rb:168:in `start_input'"], :level=>:warn}

The port is being used by Logstash, as shown by

sudo lsof -nPi :5000


COMMAND  PID     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    6837 logstash   13u  IPv6 1260946      0t0  UDP *:5000 
java    6837 logstash   16u  IPv6 1260947      0t0  TCP *:5000 (LISTEN)

I've tried reading up on this ticket https://github.com/elastic/logstash/issues/1896 but I don't think this is a reserved port issue or a permissions issue.

Could anyone help me debug this error? From what I can tell, my configuration is correct.

[jayaramsankara]

I am seeing the same issue. The exception is :
message=>"UDP listener died", :exception=>#, :backtrace=>["/var/vcap/packages/logstash/lib/logstash/inputs/udp.rb:80:in udp_listener'", "org/jruby/RubyKernel.java:1521:inloop'", "/var/vcap/packages/logstash/lib/logstash/inputs/udp.rb:78:in udp_listener'", "/var/vcap/packages/logstash/lib/logstash/inputs/udp.rb:50:inrun'", "/var/vcap/packages/logstash/lib/logstash/pipeline.rb:163:in inputworker'", "/var/vcap/packages/logstash/lib/logstash/pipeline.rb:157:instart_input'"]

I am not using docker though. This is with log stash running on ubuntu.

[Wurzelmann]

I've got the same problem:

{:timestamp=>"2015-06-15T11:54:24.265000+0200", :message=>"syslog listener died", :protocol=>:udp, :address=>"0.0.0.0:5517", :exception=>#, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:138:in udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:117:inserver'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}

Using multiple syslog inputs at once, tried to replace "syslog" input with "tcp", did not work either.

[Wurzelmann]

Okay, maybe this helps:

I changed a line in the syslog input gem file (/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb in my case). There's a part, where the TCP listener is defined:

def tcp_listener(output_queue)
@logger.info("Starting syslog tcp listener", :address => "#{@host}:#{@port}")
@tcp = TCPServer.new(@host, @port)

loop do
  socket = @tcp.accept
  @tcp_sockets << socket

  break if @shutdown_requested.true?

  Thread.new(output_queue, socket) do |output_queue, socket|
    tcp_receiver(output_queue, socket)
  end 
end 

ensure
close_tcp
end # def tcp_listener

I replaced " break if @shutdown_requested.true?" with " break if @shutdown_requested.false?" and logstash hasn't crashed since then. This was three days ago, normally logstash would crash after a few hours at the latest. It is a dirty hack, but it seems to work...

[stith]

I'm running into this same problem. Added a comment on logstash-plugins/logstash-input-syslog#18 since this is an issue with the plugin, not logstash itself.

[machzqcq]

I am getting the same error by listening to the local syslog on the same server that logstash is present. Hence seeing errors on port 514 (default port that syslog writes to). Anybody has a workaround please?

[ichandan16]

Getting following error while stopping and starting logstash server:

UDP listener died {:exception=>#, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-0.1.4/lib/logstash/inputs/udp.rb:79:in udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-0.1.4/lib/logstash/inputs/udp.rb:49:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:176:in inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.1-java/lib/logstash/pipeline.rb:170:instart_input'"], :level=>:warn}

[jotunskij]

Same problem here. Running docker with logstash:latest.

input {
  udp {
    port => 5000
    codec => json
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
  }
}

{:timestamp=>"2016-03-23T07:36:21.715000+0000", :message=>"UDP listener died", :exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3682:in `select'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:331:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:325:in `start_input'"], :level=>:warn}

[davidt99]

@jotunskij Had the same issue, and found out that docker opens ports on TCP by default unless UDP is explicitly mentioned: -p 5959:5959 (that's open TCP port) -p 5959:5959/udp (that's open UDP)

[jotunskij]

@davidt99 Unfortunately I already published the port with /udp specified. However I did seem to solve this problem by just switching the port and binding to 0.0.0.0. This is the complete flag I'm using with docker run:
--publish 0.0.0.0:12201:12201/udp

[houht]

Same problem here. Running docker with logstash:latest.

input {

udp {
    type => "haproxy"
    port => "514"
}

}

docker run --name=logstash_mesos \
--net="host" \
--publish=0.0.0.0:514:514/udp \
-v /opt/logstash/conf:/config \
logstash -f /config/my.conf

WARN logstash.inputs.udp - UDP listener died {:exception=>#, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in bind'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.0.3/lib/logstash/inputs/udp.rb:67:inudp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.0.3/lib/logstash/inputs/udp.rb:50:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:331:ininputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:325:in `start_input'"]

[houht]

but 5140 is OK：

input {

udp {
    type => "haproxy"
    port => "5140"
}

}

02:38:21.137 [[main] 02:38:21.496 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://~hidden~:[email protected]:9200"]}}
02:38:21.500 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Using mapping template from {:path=>"/json/logstash_haproxy.json"}

[IrlJidel]

what user are you running as? To bind to 514 (or any port below 1024) you need to run as root

[houht]

@IrlJidel Thank you, you're right. I did't run as root .

[vincenzodnp]

@jotunskij Had the same issue, and found out that docker opens ports on TCP by default unless UDP is explicitly mentioned: -p 5959:5959 (that's open TCP port) -p 5959:5959/udp (that's open UDP)

I solved by publishing the udp port. +1