Logstash: Service fails to stop (v1.5.0.rc2)
$ sudo service logstash stop
Killing logstash (pid 10236) with SIGTERM
Waiting logstash (pid 10236) to die...
Waiting logstash (pid 10236) to die...
Waiting logstash (pid 10236) to die...
Waiting logstash (pid 10236) to die...
Waiting logstash (pid 10236) to die...
logstash stop failed; still running.
any idea what could be the cause?
$ sudo /opt/logstash/bin/logstash agent --version --debug --verbose
logstash 1.5.0.rc2
jruby 1.7.17 (1.9.3p392) 2014-12-09 fafd1a7 on OpenJDK 64-Bit Server VM 1.7.0_75-b13 +jit [linux-amd64]
java 1.7.0_75 (Oracle Corporation)
jvm OpenJDK 64-Bit Server VM / 24.75-b04
Elasticsearch: Version: 1.3.7, Build: 3042293/2014-12-16T13:59:32Z, JVM: 1.7.0_75
gem addressable 2.3.7
gem atomic 1.1.99
gem avl_tree 1.2.1
gem awesome_print 1.6.1
gem json 1.8.2
gem nokogiri 1.6.6.2
gem aws-sdk-v1 1.61.0
gem aws-sdk 1.61.0
gem thread_safe 0.3.4
gem descendants_tracker 0.0.4
gem ice_nine 0.11.1
gem axiom-types 0.1.1
gem backports 3.6.4
gem bindata 2.1.0
gem buftok 0.2.0
gem bundler 1.8.4
gem cabin 0.7.1
gem cinch 2.2.4
gem clamp 0.6.4
gem coderay 1.1.0
gem coercible 1.0.0
gem concurrent-ruby 0.8.0
gem edn 1.0.6
gem multi_json 1.11.0
gem elasticsearch-api 1.0.7
gem multipart-post 2.0.0
gem faraday 0.9.1
gem elasticsearch-transport 1.0.7
gem elasticsearch 1.0.8
gem equalizer 0.0.9
gem ffi 1.9.6
gem ffi-rzmq 1.0.0
gem minitar 0.5.4
gem file-dependencies 0.1.6
gem filesize 0.0.4
gem filewatch 0.6.1
gem http_parser.rb 0.6.0
gem ftw 0.0.42
gem gelf 1.3.2
gem gelfd 0.2.0
gem geoip 1.4.0
gem gmetric 0.1.3
gem hitimes 1.2.2
gem http 0.6.3
gem i18n 0.6.9
gem jar-dependencies 0.1.7
gem jls-grok 0.11.1
gem jls-lumberjack 0.0.22
gem jrjackson 0.2.8
gem jruby-httpclient 1.1.1
gem virtus 1.0.4
gem maven-tools 1.0.7
gem ruby-maven-libs 3.1.1
gem ruby-maven 3.1.1.0.8
gem jruby-kafka 1.1.2
gem jruby-win32ole 0.8.5
gem mime-types 2.4.3
gem method_source 0.8.2
gem slop 3.6.0
gem spoon 0.0.4
gem pry 0.10.1
gem rack 1.6.0
gem rack-protection 1.5.3
gem tilt 1.4.1
gem sinatra 1.4.5
gem stud 0.0.19
gem polyglot 0.3.5
gem treetop 1.4.15
gem logstash-core 1.5.0.rc2
gem logstash-codec-collectd 0.1.7
gem logstash-codec-dots 0.1.5
gem logstash-codec-edn 0.1.5
gem logstash-codec-line 0.1.5
gem logstash-codec-edn_lines 0.1.6
gem logstash-codec-es_bulk 0.1.5
gem msgpack-jruby 1.4.1
gem logstash-codec-fluent 0.1.5
gem logstash-codec-graphite 0.1.5
gem logstash-codec-json 0.1.6
gem logstash-codec-json_lines 0.1.6
gem logstash-codec-msgpack 0.1.6
gem logstash-patterns-core 0.1.6
gem logstash-codec-multiline 0.1.6
gem logstash-codec-netflow 0.1.4
gem logstash-codec-oldlogstashjson 0.1.5
gem logstash-codec-plain 0.1.5
gem logstash-codec-rubydebug 0.1.6
gem murmurhash3 0.1.6
gem logstash-filter-anonymize 0.1.4
gem logstash-filter-checksum 0.1.5
gem logstash-filter-clone 0.1.4
gem logstash-filter-csv 0.1.4
gem logstash-input-generator 0.1.3
gem logstash-output-null 0.1.3
gem logstash-filter-date 0.1.5
gem logstash-filter-dns 0.1.4
gem logstash-filter-drop 0.1.4
gem logstash-filter-fingerprint 0.1.4
gem logstash-filter-geoip 0.1.7
gem logstash-filter-grok 0.1.8
gem logstash-filter-json 0.1.4
gem logstash-filter-kv 0.1.5
gem metriks 0.9.9.7
gem logstash-filter-metrics 0.1.6
gem logstash-filter-mutate 0.1.5
gem logstash-filter-multiline 0.1.5
gem logstash-filter-ruby 0.1.4
gem logstash-filter-sleep 0.1.4
gem logstash-filter-split 0.1.5
gem logstash-filter-syslog_pri 0.1.4
gem logstash-filter-throttle 0.1.4
gem logstash-filter-urldecode 0.1.4
gem user_agent_parser 2.2.0
gem logstash-filter-useragent 0.1.6
gem logstash-filter-uuid 0.1.4
gem xml-simple 1.1.5
gem logstash-filter-xml 0.1.4
gem logstash-input-couchdb_changes 0.1.3
gem logstash-input-elasticsearch 0.1.4
gem logstash-input-eventlog 0.1.4
gem logstash-input-exec 0.1.3
gem logstash-input-file 0.1.6
gem logstash-input-ganglia 0.1.3
gem logstash-input-gelf 0.1.3
gem logstash-input-tcp 0.1.3
gem logstash-input-graphite 0.1.3
gem mail 2.6.3
gem logstash-input-imap 0.1.3
gem logstash-input-irc 0.1.3
gem logstash-input-kafka 0.1.11
gem logstash-input-log4j 0.1.3
gem logstash-input-lumberjack 0.1.4
gem logstash-input-pipe 0.1.4
gem march_hare 2.5.1
gem logstash-input-rabbitmq 0.1.3
gem redis 3.2.1
gem logstash-input-redis 0.1.3
gem logstash-mixin-aws 0.1.7
gem logstash-input-s3 0.1.8
gem snmp 1.2.0
gem logstash-input-snmptrap 0.1.3
gem logstash-input-sqs 0.1.3
gem logstash-input-stdin 0.1.3
gem logstash-input-syslog 0.1.3
gem memoizable 0.4.2
gem naught 1.0.0
gem simple_oauth 0.3.1
gem twitter 5.12.0
gem logstash-input-twitter 0.1.5
gem logstash-input-udp 0.1.3
gem logstash-input-unix 0.1.3
gem xmpp4r 0.5
gem logstash-input-xmpp 0.1.3
gem logstash-input-zeromq 0.1.3
gem tzinfo 1.2.2
gem rufus-scheduler 2.0.24
gem logstash-output-cloudwatch 0.1.3
gem logstash-output-file 0.1.5
gem logstash-output-csv 0.1.3
gem manticore 0.3.5
gem logstash-output-elasticsearch 0.1.18
gem logstash-output-email 0.1.4
gem logstash-output-exec 0.1.3
gem logstash-output-ganglia 0.1.3
gem logstash-output-gelf 0.1.3
gem logstash-output-graphite 0.1.6
gem logstash-output-hipchat 0.1.3
gem logstash-output-http 0.1.3
gem logstash-output-irc 0.1.3
gem logstash-output-juggernaut 0.1.3
gem logstash-output-kafka 0.1.8
gem logstash-output-lumberjack 0.1.4
gem logstash-output-nagios 0.1.3
gem logstash-output-nagios_nsca 0.1.3
gem logstash-output-opentsdb 0.1.3
gem logstash-output-pagerduty 0.1.3
gem logstash-output-pipe 0.1.3
gem logstash-output-rabbitmq 0.1.4
gem logstash-output-redis 0.1.3
gem logstash-output-s3 0.1.6
gem logstash-output-sns 0.1.3
gem logstash-output-sqs 0.1.3
gem statsd-ruby 1.2.0
gem logstash-output-statsd 0.1.5
gem logstash-output-stdout 0.1.4
gem logstash-output-tcp 0.1.4
gem logstash-output-udp 0.1.4
gem logstash-output-xmpp 0.1.3
gem logstash-output-zeromq 0.1.4
using strace:
$ sudo strace service logstash stop
execve("/usr/sbin/service", ["service", "logstash", "stop"], [/* 21 vars */]) = 0
brk(0) = 0x2419000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6c9e625000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=35317, ...}) = 0
mmap(NULL, 35317, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f6c9e61c000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\30\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1811128, ...}) = 0
mmap(NULL, 3925176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6c9e046000
mprotect(0x7f6c9e1fa000, 2097152, PROT_NONE) = 0
mmap(0x7f6c9e3fa000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b4000) = 0x7f6c9e3fa000
mmap(0x7f6c9e400000, 17592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6c9e400000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6c9e61b000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6c9e61a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6c9e619000
arch_prctl(ARCH_SET_FS, 0x7f6c9e61a700) = 0
mprotect(0x7f6c9e3fa000, 16384, PROT_READ) = 0
mprotect(0x619000, 4096, PROT_READ) = 0
mprotect(0x7f6c9e627000, 4096, PROT_READ) = 0
munmap(0x7f6c9e61c000, 35317) = 0
getpid() = 22456
rt_sigaction(SIGCHLD, {0x40f100, ~[RTMIN RT_1], SA_RESTORER, 0x7f6c9e07c150}, NULL, 8) = 0
geteuid() = 0
brk(0) = 0x2419000
brk(0x243a000) = 0x243a000
getppid() = 22455
getcwd("/home/smueller", 4096) = 15
open("/usr/sbin/service", O_RDONLY) = 3
fcntl(3, F_DUPFD, 10) = 10
close(3) = 0
fcntl(10, F_SETFD, FD_CLOEXEC) = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x40f100, ~[RTMIN RT_1], SA_RESTORER, 0x7f6c9e07c150}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7f6c9e07c150}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7f6c9e07c150}, NULL, 8) = 0
read(10, "#!/bin/sh\n\n#####################"..., 8192) = 4768
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6c9e61a9d0) = 22457
close(4) = 0
read(3, "service\n", 128) = 8
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22457
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22457
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6c9e61a9d0) = 22458
close(4) = 0
read(3, "service\n", 128) = 8
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22458
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22458
chdir("/") = 0
faccessat(AT_FDCWD, "/etc/init/logstash.conf", R_OK) = -1 ENOENT (No such file or directory)
geteuid() = 0
stat("/etc/init.d/logstash", {st_mode=S_IFREG|0755, st_size=3462, ...}) = 0
faccessat(AT_FDCWD, "/etc/init.d/logstash", X_OK) = 0
execve("/usr/local/sbin/env", ["env", "-i", "LANG=en_US.UTF-8", "PATH=/usr/local/sbin:/usr/local/"..., "TERM=xterm-256color", "/etc/init.d/logstash", "stop"], [/* 23 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/env", ["env", "-i", "LANG=en_US.UTF-8", "PATH=/usr/local/sbin:/usr/local/"..., "TERM=xterm-256color", "/etc/init.d/logstash", "stop"], [/* 23 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/sbin/env", ["env", "-i", "LANG=en_US.UTF-8", "PATH=/usr/local/sbin:/usr/local/"..., "TERM=xterm-256color", "/etc/init.d/logstash", "stop"], [/* 23 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/bin/env", ["env", "-i", "LANG=en_US.UTF-8", "PATH=/usr/local/sbin:/usr/local/"..., "TERM=xterm-256color", "/etc/init.d/logstash", "stop"], [/* 23 vars */]) = 0
brk(0) = 0x19ec000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa930e84000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=35317, ...}) = 0
mmap(NULL, 35317, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa930e7b000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\30\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1811128, ...}) = 0
mmap(NULL, 3925176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fa9308a5000
mprotect(0x7fa930a59000, 2097152, PROT_NONE) = 0
mmap(0x7fa930c59000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b4000) = 0x7fa930c59000
mmap(0x7fa930c5f000, 17592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fa930c5f000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa930e7a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa930e79000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa930e78000
arch_prctl(ARCH_SET_FS, 0x7fa930e79700) = 0
mprotect(0x7fa930c59000, 16384, PROT_READ) = 0
mprotect(0x605000, 4096, PROT_READ) = 0
mprotect(0x7fa930e86000, 4096, PROT_READ) = 0
munmap(0x7fa930e7b000, 35317) = 0
brk(0) = 0x19ec000
brk(0x1a0d000) = 0x1a0d000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3165552, ...}) = 0
mmap(NULL, 3165552, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa9305a0000
close(3) = 0
execve("/etc/init.d/logstash", ["/etc/init.d/logstash", "stop"], [/* 3 vars */]) = 0
brk(0) = 0x204a000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe707ea8000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=35317, ...}) = 0
mmap(NULL, 35317, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe707e9f000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\30\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1811128, ...}) = 0
mmap(NULL, 3925176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe7078c9000
mprotect(0x7fe707a7d000, 2097152, PROT_NONE) = 0
mmap(0x7fe707c7d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b4000) = 0x7fe707c7d000
mmap(0x7fe707c83000, 17592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe707c83000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe707e9e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe707e9d000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe707e9c000
arch_prctl(ARCH_SET_FS, 0x7fe707e9d700) = 0
mprotect(0x7fe707c7d000, 16384, PROT_READ) = 0
mprotect(0x619000, 4096, PROT_READ) = 0
mprotect(0x7fe707eaa000, 4096, PROT_READ) = 0
munmap(0x7fe707e9f000, 35317) = 0
getpid() = 22456
rt_sigaction(SIGCHLD, {0x40f100, ~[RTMIN RT_1], SA_RESTORER, 0x7fe7078ff150}, NULL, 8) = 0
geteuid() = 0
brk(0) = 0x204a000
brk(0x206b000) = 0x206b000
getppid() = 22455
getcwd("/", 4096) = 2
open("/etc/init.d/logstash", O_RDONLY) = 3
fcntl(3, F_DUPFD, 10) = 10
close(3) = 0
fcntl(10, F_SETFD, FD_CLOEXEC) = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x40f100, ~[RTMIN RT_1], SA_RESTORER, 0x7fe7078ff150}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7fe7078ff150}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7fe7078ff150}, NULL, 8) = 0
read(10, "#!/bin/sh\n# Init script for logs"..., 8192) = 3462
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22459
close(4) = 0
read(3, "0\n", 128) = 2
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22459
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22459
faccessat(AT_FDCWD, "/etc/default/logstash", R_OK) = 0
open("/etc/default/logstash", O_RDONLY) = 3
fcntl(3, F_DUPFD, 10) = 11
close(3) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
read(11, "###############################\n"..., 8192) = 779
read(11, "", 8192) = 0
close(11) = 0
faccessat(AT_FDCWD, "/etc/sysconfig/logstash", R_OK) = -1 ENOENT (No such file or directory)
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22460
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22460
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22460
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22461
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22461
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22461
write(1, "Killing logstash (pid 21877) wit"..., 42Killing logstash (pid 21877) with SIGTERM
) = 42
kill(21877, SIGTERM) = 0
write(1, "Waiting logstash (pid 21877) to "..., 39Waiting logstash (pid 21877) to die...
) = 39
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22463
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22463
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22463
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
stat("/sbin/sleep", 0x7fffea40fae0) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/sleep", 0x7fffea40fae0) = -1 ENOENT (No such file or directory)
stat("/bin/sleep", {st_mode=S_IFREG|0755, st_size=27024, ...}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22464
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22464
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22464
write(1, "Waiting logstash (pid 21877) to "..., 39Waiting logstash (pid 21877) to die...
) = 39
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22477
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22477
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22477
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22478
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22478
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22478
write(1, "Waiting logstash (pid 21877) to "..., 39Waiting logstash (pid 21877) to die...
) = 39
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22567
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22567
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22567
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22568
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22568
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22568
write(1, "Waiting logstash (pid 21877) to "..., 39Waiting logstash (pid 21877) to die...
) = 39
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22625
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22625
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22625
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22626
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22626
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22626
write(1, "Waiting logstash (pid 21877) to "..., 39Waiting logstash (pid 21877) to die...
) = 39
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22702
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22702
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22702
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22703
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22703
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22703
stat("/var/run/logstash.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe707e9d9d0) = 22749
close(4) = 0
read(3, "21877\n", 128) = 6
read(3, "", 128) = 0
close(3) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22749
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11) = 22749
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(1, F_DUPFD, 10) = 11
close(1) = 0
fcntl(11, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 1) = 1
close(3) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fcntl(2, F_DUPFD, 10) = 12
close(2) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
dup2(3, 2) = 2
close(3) = 0
kill(21877, SIG_0) = 0
dup2(11, 1) = 1
close(11) = 0
dup2(12, 2) = 2
close(12) = 0
write(1, "logstash stop failed; still runn"..., 37logstash stop failed; still running.
) = 37
exit_group(0) = ?
svenmueller
All 31 comments
Same here. Ubuntu 14.04, Logstash v1.5.0.rc2
MarkGavalda
on 9 Mar 2015
@svenmueller and @MarkGavalda can you please include your config file or at least the list of inputs/outputs you are using.
wiibaa
on 9 Mar 2015
So there's two problems:
1) THe stop on the init script sends SIGTERM which tells Logstash to shutdown safely. If it doesn't finish shutting down within a few seconds, no further action is taken.
2) Logstash was told to shutdown, but it's not, for some reason. Need logs to know more.
For the init script behavior, I"m open to adding a kill command (/etc/init.d/logstash kill) that will cause immediate and unsafe termination of the Logstash process. Is this desirable?
jordansissel
on 9 Mar 2015
I'm against adding some non-standard command which isn't compliant with any modern init system. If user requested to stop, then he know what he does and it's not OK to bail out. Just initiate kill after long enough delay, usually 30 seconds.
z0rc
on 9 Mar 2015
@wiibaa here is the list of inputs/outputs i'm using:
inputs
execlumberjacktcppipe
outputs:
elasticsearch
(see the full config here https://gist.github.com/svenmueller/4faac33ac051263f1f96)
i agree with @z0rc, adding a kill command ist not very helpful here. we should find out why the stop fails instead.
svenmueller
on 9 Mar 2015
I found that this is related to https://github.com/elasticsearch/logstash/issues/2798
I can usually stop it with the init.d script, however when logstash hangs (as I described in the linked issue) then this stopping problem comes up and only kill -9 helps.
MarkGavalda
on 9 Mar 2015
(Btw, i'm using Ubuntu 12.04)
svenmueller
on 9 Mar 2015
This kind of behavior usually occurs when the output is slowing down and exerting backpressure. So when logstash isn't able to output events quickly enough, it will fill up it's small (20 items) queue and stop consuming events.
Right now, the exit strategy for logstash consists of grabbing sigterm, producing a shutdownsignal event and sending it through itself in order to make all plugins shutdown. The problem is that, if Logstash is exerting backpressure, the queues will be full so the shutdown signal is queue like any other event. This prevents logstash from ever terminating gracefully :(
Are you seeing any messages on the elasticsearch side related to slow indexing operations?
jsvd
on 9 Mar 2015
We have this issue as well. Is there any way to throw more resources at it? Perhaps tweak a setting to allow it to better handle these requests? I agree with jsvd's assessment because when logstash is really busy, eventually it just dies. I put a hack in place that checks to see if it is running every 2 minutes and restarts if dead. When it gets past a really busy period, everything hums along. A few notes about our configuration: mostly jenkins console output via plugin (huge) -> redis input -> logstash -> elasticsearch
radkin
on 11 Mar 2015
I have also this issue on Centos 6.x 64 bits, with JDK 1.7.0-55 and logstash-1.5.0.rc2-1
Inputs: syslog, netflow, lumberjack
Outputs: Elasticsearch, Email
Only kill -9 stop the process.
sbarrier
on 12 Mar 2015
Experiencing same thing. Centos 6.x , java 1.7.0.71, logstash-1.5.0.rc2
allenmchan
on 18 Mar 2015
If user requested to stop, then he know what he does and it's not OK to bail out. Just initiate kill after
@z0rc I disagree. "stop" doesn't imply "force an unsafe shutdown by terminating the process". kill -9 on a process is equivalent to pulling the power cord from your servers (instead of, say, 'shutdown' command)
jordansissel
on 18 Mar 2015
Logstash failing to stop when requested is a symptom of Logstash being stuck and unable to terminate without losing messages. In my opinion, the default behavior of "stop" should _not_ cause data loss.
jordansissel
on 18 Mar 2015
I put a hack in place that checks to see if it is running every 2 minutes and restarts if dead
@radkin I'm confused. How is (/etc/init.d/logstash stop not completing its shutdown) related to (restart if logsatsh dies) ?
Also, "restart when dead" is not a hack. Restarting dead services is a _feature_ of upstart, systemd, runit, daemontools, supervisord, and many more service managers.
jordansissel
on 18 Mar 2015
As a workaround, y'all can modify the init script to use KILL instead of TERM.
jordansissel
on 18 Mar 2015
we should find out why the stop fails instead.
@svenmueller Termination is blocked waiting for in-flight events to flush. If you interrupt this (via kill -KILL), you will lose events. This is why the script doesn't just give up and do a kill -KILL because no default behavior in Logstash (or its service management) should explicitly request data loss.
jordansissel
on 18 Mar 2015
Short term proposals:
1) Add a kill command to the init script that uses kill -KILL instead of kill -TERM
2) or, make stop loop until the process successfully dies after sending SIGTERM
Long term:
Logstash will soon support in-flight message persistence. Once this happens, kill -KILL will not result in data loss of in-flight messages (assuming the input protocols support acknowledgements). When we get to this point, I will consider a default behavior that tries a TERM followed by a KILL if that doesn't work.
jordansissel
on 18 Mar 2015
Is it a red herring that logstash stops fine on 1.4.2 and started having issues in 1.5 rc2?
allenmchan
on 18 Mar 2015
@jordansissel our issue is no longer with stopping logstash, it is with keeping it from dying. Sorry I didn't make it clear that the "hack" script I put in place is to keep it alive. Our service manager is not handling logstash when it dies, although I agree that we could have it do this. Since our issue may not be relevant to this bug, please disregard. I will gather more information and either post on another issue or create one. It turns out, our issue was from the influxdb plugin; an issue that was fixed 20 days ago with epok. it's fixed
radkin
on 18 Mar 2015
@jsvd can you check if this is an issue with the specific inputs not being able to be stopped?
suyograo
on 18 Mar 2015
@jordansissel wrote:
Short term proposals:
1) Add a kill command to the init script that uses kill -KILL instead of kill -TERM
Won't help on distributions that use Upstart though, right?
2) or, make stop loop until the process successfully dies after sending SIGTERM
Are we sufficiently confident that plugins will behave correctly and honor shutdown requests so that we can trust hanging stop requests to be genuine? Logstash doesn't have the best track record in this regard (I note that elastic/logstash#1779 is still open).
I'd prefer allowing a reasonable time for the graceful shutdown but eventually issue a SIGKILL.
magnusbaeck
on 19 Mar 2015
Won't help on distributions that use Upstart though, right?
Upstart sends sigterm also, as does systemd (I think). These may be configurable, though, I don't think it does "Try to ask nicely for a termination, then kill the process permanently"
I don't want to conflate "stop" with "under any circumstances, please kill the process" because there could be problems that a human would be able to intervene and save that would otherwise cause permanent message loss today.
Once we have persistence of events in @colinsurprenant's queue branch, we can consider an external shutdown signal that will give up and SIGKILL after a time. However, until we have that, I cannot support any automatic shutdown that kills Logstash in this way.
Logstash doesn't have the best track record in this regard
We should fix the bugs, not make default behaviors that work around the bugs. :)
jordansissel
on 19 Mar 2015
the shutdown stall regression is probably related to #2863 - @jsvd can you review my proposed commit?
colinsurprenant
on 20 Mar 2015
^^ is now included in the #2869 WIP PR
colinsurprenant
on 20 Mar 2015
suyograo
on 6 Apr 2015
not solved for me with RC3 (https://github.com/elastic/logstash/issues/3165)
svenmueller
on 5 May 2015
Dropping a comment in here for posterity. An unintended consequence of this behavior (holding on to the shutdown event until the pipeline is flushed) is that it can give the impression that Logstash "crashed" without any logging as to why.
Consider this scenario:
- Logstash is up, but output is unavailable (whether before or after initial startup of the daemon)
- Someone (or something, e.g.: Ansible) issues a systemctl logstash stop or service logstash stop, etc
- As discussed above, Logstash will not honor that request in an attempt to prevent data loss, resulting in the shutdown request being put on hold, pending a flush of the output's queue
- Once access to the output is restored, the queue is flushed and the shutdown request is finally processed
- At this point, the process "inexplicably" stops, which would give the impression that Logstash seemingly crashed for no reason
The "best" scenario I can think of would hinge on the delivery of message persistence. At that point, we can safely kill - if needed - the daemon.
Another alternative would be to have the .since_db files only updated upon successful delivery to the output(s), but I presume that would be really challenging to manage given that not all messages would necessarily go to all outputs.
Anyway, hope that helps someone's sanity in the future :)
cjchand
on 30 Jun 2016
solved on ubuntu with change in init script -TERM to -9.
ka4a
on 15 Jul 2016
I don't think a SIGKILL is a good idea for this at all. My buddy created a script to restart Logstash when it reaches a stuck output state like mentioned in this thread, and I have experienced data loss :( :( :( He used a kill -9 in his script if the restart is unsuccessful.
ghost
on 11 Oct 2016
@chromechris1 please see my comment here - https://github.com/elastic/logstash/issues/2796#issuecomment-82738875 - a stalled shutdown is a nearly always a symptom of a traffic jam in your data flow (inputs to outputs), and logstash is still running after SIGTERM because it wants to finish delivering some in-flight data before it shuts down completely. SIGKILL is fine if you want Logstash to die and can accept data loss, otherwise, you'll need to wait for Logstash to finish delivering and/or figure out what is causing Logstash to continue having undelivered messages (some downstream output is broken somehow, for example.)
jordansissel
on 11 Oct 2016
@jordansissel Awsome! I am wondering what can cause an input/filter & output thread to get stuck. Is there any ressources/documentation you recommend for learning how the internal workings of these Logstash threads? Thanks in advance.
chromechris
on 29 Oct 2016
Related issues
behkxyz
路
3Comments
cschotke
路
3Comments
jakelandis
路
4Comments
ashangit
路
4Comments
bobbyhubbard
路
3Comments
Most helpful comment
I have also this issue on Centos 6.x 64 bits, with JDK 1.7.0-55 and logstash-1.5.0.rc2-1
Inputs: syslog, netflow, lumberjack
Outputs: Elasticsearch, Email
Only kill -9 stop the process.