Logstash: Logstash 5.0 package does not provide / create initscript

[widhalmt]

Hi,

I just upgraded one of my test hosts from Logstash 2.2. to Logstash 5.0. After the upgrade I had no Init script.

Running /usr/share/logstash/bin/system-install gives me the following output:

Sending all logs to /tmp/tmp.blXM2X3JuF
Successfully created system startup script for Logstash

But I still have no init script. The mentioned logfile in /tmp does not exist.

OS: CentOS 6.8 x86_64
Installation: logstash-5.0.0-1.noarch from Elastic repositories
SELinux: permissive

I had several other init scripts for running an extra shipper instance on the host and some tests I did with special Logstash setups. But /etc/init.d/logstash was deleted during the upgrade and it seems, it's not coming back.

Is there any extra information I can provide you with?

Cheers,
Thomas

[widhalmt]

It's not working when providing the startup.options file, either

/usr/share/logstash/bin/system-install /etc/logstash/startup.options

[widhalmt]

Digging further into what the script does shows, that it is indeed creating files for starting logstash but for upstart. Since I'm using SysV init scripts on this system (upstart is installed), I'm wondering if the detection of the init system is working as it should be.

[jordansissel]

The system-install tool will detect if upstart is available and use it. On upstart systems, you need to use initctl [command] logstash where command is start, stop, restart, etc.

[ncastrocosta]

I have the same problem.
I installed from apt repository following the docs (https://www.elastic.co/guide/en/logstash/current/installing-logstash.html).
OS: Ubuntu 16.04

[jordansissel]

There is no init script provided by logstash package on Upstart or systemd systems (CentOS 6 or 7, Ubuntu 08.10 and upward)

[jordansissel]

If you are running on an upstart system and still want a shell script, this will work:

#!/bin/sh

echo "This job runs via upstart, invoking upstart now..."
exec initctl "$1" logstash

[ncastrocosta]

Hi, @jordansissel.

The doc says to "configure Logstash to automatically start during bootup" I should run "sudo update-rc.d logstash defaults 95 10", but in this moment I receive the error message "update-rc.d: error: initscript does not exist: /etc/init.d/logstash". That's the reason I expected the init script.

When I run "initctl start logstash" I get nothing. However, I must tell you I'm running inside a docker instance. Would it make any difference?

Thanks!

[jordansissel]

@ncastrocosta The context around that command is this:

If your distribution is using SysV init, then you will need to run:

sudo update-rc.d logstash defaults 95 10

The key phrase there is "If your distribution is using sysv init", and, from the report here, I would guess that your distribution is not using sysv init, but using upstart or systemd.

[jordansissel]

However, I must tell you I'm running inside a docker instance. Would it make any difference?

Probably this is theproblem. We don't currently have documentation for running in Docker.

I would recommend you invoke logstash from Docker directly. If you install via rpm or deb, use /usr/share/logstash/bin/logstash ... as your command

[jordansissel]

@ncastrocosta It is possible that the rpm/dpkg install is detecting incorrectly that your OS uses systemd or upstart. We do not currently do not test our rpm or deb packages within docker, so I don't have any expectations to set for you, sorry.

I'd recommend not using rpms or debs for Docker, at least for now, and using the tarball or zip we release, as it may be more suited to your scenario.

[ncastrocosta]

@jordansissel
yes, I'll give a try the tarball release.
Thanks!

[jordansissel]

@widhalmt Any ideas on a fix? I believe the system-install tool is doing the correct thing by not installing a sysv init script when your os is using upstart.

Should the system-install tool tell you what it did more clearly? Specifically, maybe it could output this:

Successfully created system Upstart job for Logstash. Run initctl start logstash to start this.

Thoughts?

[jordansissel]

For some history, the library Logstash uses to generate upstart/systemd/etc is https://github.com/jordansissel/pleaserun. This library used to generate an /etc/init.d wrapper for Upstart jobs. However, a bug was found with how chkconfig attempts to use init scripts, and so, to resolve that bug, I removed the init.d wrapper.

If I recall correctly, chkconfig configures under which run level a service is started. Upstart already has a feature for controlling the start and stop runlevels. This means upstart and chkconfig probably not compatible things because both will have different ideas of when a service should be started.

Given the background and details above, I believe it is correct that, when delivering an Upstart job, we do not create an /etc/init.d wrapper script.

[widhalmt]

@jordansissel Yes, telling the user what the script did more clearly would be benefit. It took me some time to find that it created an upstart file instead.

Maybe the script should use stat /proc/1/exe to find if upstart is really the leading system und place the wrapper if it isn't?

[untergeek]

@widhalmt Unfortunately, /proc/1/exe has proven an unreliable source of authority, as some systems use a hybrid approach. We'll see what we can do, however.

[superawesome]

Just struggled through this myself. RHEL6 does indeed have Upstart, but virtually all services (including Elastic.co ES and Kibana packages, as well as stock RH packages like Apache and MySQL) actually use /etc/init.d/ scripts... not /etc/init/ Upstart configs. RHEL7 uses systemd, so at least it's unified in the future... but that's just a consolation prize for our current infra. :(

This is the only app I know of that actually uses Upstart on RHEL6. As such it breaks the "de facto standard" service/chkconfig pattern on that platform. It's also a change from previous Logstash versions on this platform, which used SysV scripts.

Also also, it's not mentioned in the "Breaking Changes" doc (https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html) or the Upgrading doc (https://www.elastic.co/guide/en/logstash/5.0/upgrading-logstash-5.0.html), so after upgrading it's just "gone". Looking for it as an Upstart service was definitely not on my radar. :)

If the system-install script just output which type of startup script it made and where it put it (and/or how to start the service), that would definitely have saved us some time troubleshooting. As-is, even running the script under "bash -x" was unrevealing... it had to be edited to preserve the tmp log file, which contained the helpful information.

[superawesome]

I noticed that the log file has this in it:

:message=>"No platform selected. Autodetecting..."

... Is there a way to tell it what platform to generate for? I don't see any such setting documented anywhere (and system-install has no --help or manpage) but it would probably go a long way toward helping RHEL/CentOS6 folks.

[jamesrenfro]

I ran into this problem as well, and would definitely appreciate some way to encourage system-install not to use upstart. The current behavior is particularly confusing because it's different from the way that Filebeat, ES, and Kibana work on install.

[untergeek]

If you want to hack the system-install script, you can add the -p flag:

https://github.com/jordansissel/pleaserun/blob/master/lib/pleaserun/cli.rb#L15

You may also need the -v flag, just beneath that.

[jamesrenfro]

Great - thanks

[suyograo]

Docs fixed in #6151

Tracking issue https://github.com/elastic/logstash/issues/6209

[marcindulak]

The problem is still present on CentOS7. The instructions at
https://www.elastic.co/guide/en/logstash/current/running-logstash.html#running-logstash-systemd say

 Logstash places the systemd unit files in /etc/systemd/system for both deb and rpm.
 After installing the package, you can start up Logstash with:

 sudo systemctl start logstash.service

but:

$ cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
$ rpm -q logstash
logstash-5.1.1-1.noarch
$ rpm -qi logstash | grep 'Build Date'
Build Date  : Tue 06 Dec 2016 02:18:33 PM CET
$ rpm -ql logstash | grep -E '/etc/systemd|/etc/init.d'  # empty
$ yum provides '/etc/*logstash' # empty

For comparison:

 $ rpm -ql kibana | grep -E '/etc/systemd|/etc/init.d' 
 /etc/init.d/kibana
 /etc/systemd/system/kibana.service
 $ rpm -ql elasticsearch | grep -E '/etc/systemd|/etc/init.d' 
 $ /etc/init.d/elasticsearch

[untergeek]

The ability to manually select an init style has already been merged, but will not be released until Logstash v5.2.0

[inthecloud247]

@untergeek Thanks! I was able to test and this command worked to generate a sysv script for logstash 5.4 on centos 6.9:

/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv

I also tested the status/start/stop/restart commands, and it seems to work as expected.

The new system-install command has the following documentation:

# /usr/share/logstash/bin/system-install --help
Usage: system-install [OPTIONSFILE] [STARTUPTYPE] [VERSION]

NOTE: These arguments are ordered, and co-dependent

OPTIONSFILE: Full path to a startup.options file
OPTIONSFILE is required if STARTUPTYPE is specified, but otherwise looks first
in /usr/share/logstash/config/startup.options and then /etc/logstash/startup.options
Last match wins

STARTUPTYPE: e.g. sysv, upstart, systemd, etc.
OPTIONSFILE is required to specify a STARTUPTYPE.

VERSION: The specified version of STARTUPTYPE to use.  The default is usually
preferred here, so it can safely be omitted.
Both OPTIONSFILE & STARTUPTYPE are required to specify a VERSION.

For more information, see https://github.com/jordansissel/pleaserun

It probably makes sense to update the documentation to specify how to specify a startuptype:

https://www.elastic.co/guide/en/logstash/current/running-logstash.html

I've created a PR for updating the docs: https://github.com/elastic/logstash/pull/7306

[konschart]

Hello,
I was able to create an init.d logstash file using the following command

/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv

But still when i try to start the service i get unknown job message.
Any ideas ? I tried updatedb and rebooting.

[obayit]

i ran service logstash start and the service started, but I don't know if it is actually working or not.

BTW where is the logstash.conf file in ubuntu server?

[t3nebre]

i ran service logstash start and the service started, but I don't know if it is actually working or not.

BTW where is the logstash.conf file in ubuntu server?

/etc/logstash/conf.d/

[lukelee1987]

I have my logstash service running, but I cant see the ini.d file. Is that and issue? Besides that I also faced with access right issue. Does anyone encounter this?

[magicdude4eva]

On CentOS 7 I did this:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

configure repo:

cat << EOL > /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOL

and install

yum -y install logstash

Service file etc installs perfectly fine. configs are under /etc/logstash/conf.d/

[Varstahl]

I'm reporting this here in case it helps other people stuck. I tried to install Logstash on a fresh CentOS 7, but logstash.service was not installed. It appeared after uninstalling and reinstalling the same package, though.

If I'm understanding this correctly, installing the jdk after logstash causes the post-install scripts to fail resulting in an invalid or partial installation. In my case I had everything I needed installed but I inverted the order of operation.