Elastalert: Run ElastAlert as a service?

[iamrudra]

Is there a way we can add an init script for ElastAlert and run it as a service? This would be a nice enhancement to have.

[Qmando]

You could try Supervisor + one of https://github.com/Supervisor/initscripts. supervisord.conf.example has some basic default settings that should work. You may have to adjust the command to your liking.

[iamrudra]

I have certainly read about that in the documentation. This is about running it as an independent service with an init script.

[adrianlop]

@iamrudra yeah you can!

easy example (/etc/init/elastalert.conf):

start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]
exec /path_to/elastalert --params_here

same for systemd (you can stablish a relationship, so elastalert will start after elasticsearch is up):

[Unit]
Description=Elastalert
After=elasticsearch.service

[Service]
Type=simple
User=xx
Group=xx
Restart=on-failure
ExecStart=/path_to/elastalert --params_here

[Install]
WantedBy=multi-user.target

hope this helps.

[minhdanh]

@iamrudra
I made an init script on RHEL, here it is:

#!/bin/bash
# elastalert   startup script for elastalert
# pidfile:           /var/run/elastalert.pid
# chkconfig: 2345 99 01

NAME=elastalert
PIDFILE=/var/run/$NAME.pid
ELASTALERT_DIR=/opt/elastalert
ELASTALERT_USER=elastalert
CONFIG_FILE=$ELASTALERT_DIR/config.yaml
ELASTALERT=/usr/local/bin/$NAME

. /etc/rc.d/init.d/functions

case $1 in
   start)
      echo -n $"Starting $NAME: "
      cd $ELASTALERT_DIR
      daemon --user="$ELASTALERT_USER" --pidfile="$PIDFILE" "$ELASTALERT --config $CONFIG_FILE &"
      RETVAL=$?
      pid=`ps -ef | grep python | grep elastalert | awk '{print $2}'`
      if [ -n "$pid" ]; then
        echo $pid > "$PIDFILE"
      fi
   ;;
   stop)
      echo -n $"Stopping $NAME: "
      killproc -p "$PIDFILE" -d 10 "$ELASTALERT"
      RETVAL="$?"
      echo
      [ $RETVAL = 0 ] && rm -f "$PIDFILE"
   ;;
   *)
      echo "Usage: /etc/init.d/elastalert {start|stop}" ;;
esac
exit 0

[elvarb]

Here is how I got it running with systemctl

Create the service file

vi /lib/systemd/system/elastalert.service

Add this to the file

[Unit]
Description=elastalert
After=multi-user.target

[Service]
Type=simple
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/elastalert

[Install]
WantedBy=multi-user.target

Then create a link, reload the daemon, enable the service and start the service

ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service
systemctl daemon-reload
systemctl enable elastalert.service
systemctl start elastalert.service
systemctl status elastalert.service

[portante]

FWIW, this is a slightly enhanced (arguably) version of an /etc/init.d/elastalert script handling a few more common commands (restart, etc.) and capturing stderr/stdout and logging them:

#!/bin/bash
# elastalert   startup script for elastalert
# pidfile:           /var/run/elastalert.pid
# chkconfig: 2345 99 01

### BEGIN INIT INFO
# Provides: elastalert
# Required-Start: cgconfig
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: elastalert control
# Description:
### END INIT INFO

if [ $(id -u) -ne 0 ]; then
    echo "This script can be run by root only. Exiting."
    exit 4
fi

NAME=elastalert
PIDFILE=/var/run/$NAME.pid
LOCKFILE=/var/lock/subsys/$NAME
ELASTALERT_DIR=/opt/perf-dept/elastalert
ELASTALERT_USER=elastalert
CONFIG_FILE=$ELASTALERT_DIR/config.yaml
ELASTALERT=/usr/bin/$NAME

. /etc/rc.d/init.d/functions

[ -e /etc/sysconfig/$NAME ] && . /etc/sysconfig/$NAME

start() {
    [ -x $ELASTALERT ] || exit 5
    [ -f $CONFIG_FILE ] || exit 6
    echo -n $"Starting $NAME: "
    cd $ELASTALERT_DIR
    daemon --user="$ELASTALERT_USER" --pidfile="$PIDFILE" "$ELASTALERT --config $CONFIG_FILE 2>&1 | /bin/logger -p daemon.info -t $NAME > /dev/null 2>&1 &"
    retval=$?
    pid=$(ps -ef | grep python | grep $NAME | awk '{print $2}')
    if [ -n "$pid" ]; then
        echo $pid > "$PIDFILE"
    fi
    echo
    [ $retval -eq 0 ] && touch "$LOCKFILE"
    return $retval
}

stop() {
    echo -n $"Stopping $NAME: "
    killproc -p "$PIDFILE" -d 10 "$NAME"
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f "$LOCKFILE" "$PIDFILE"
    return $retval
}

restart() {
    stop
    start
}

reload() {
    restart
}

force_reload() {
    restart
}

rh_status() {
    # run checks to determine if the service is running or use generic status
    status $NAME
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}


case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?

[elastauser]

On centos 7 and tried using the script below with no luck:

#!/bin/bash
# myapp daemon
# chkconfig: 345 20 80
# description: myapp daemon
# processname: myapp

DAEMON_PATH="/elastalert/elastalert"
NAME=elastalert
DESC="elastalert"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

case "$1" in
start)
        printf "%-50s" "Starting $NAME..."
        cd $DAEMON_PATH
        daemon --user="$ELASTALERT_USER" --pidfile="$PIDFILE" "$ELASTALERT --config $CONFIG_FILE 2>&1 | /bin/logger -p daemon.info -t $NAME > /dev/null 2>&1 &"
    retval=$?
#       PID=`$DAEMON  > /dev/null 2>&1 & echo $!`
        echo $PID
       #echo "Saving PID" $PID " to " $PIDFILE
        if [ -z $PID ]; then
            printf "%s\n" "Fail"
        else
            echo $PID > $PIDFILE
            printf "%s\n" "Ok"
        fi
;;
status)
        printf "%-50s" "Checking $NAME..."
        if [ -f $PIDFILE ]; then
            PID=`cat $PIDFILE`
            if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
                printf "%s\n" "Process dead but pidfile exists"
            else
      echo "Running"
            fi
        else
            printf "%s\n" "Service not running"
        fi
;;
stop)
        printf "%-50s" "Stopping $NAME"
            PID=`cat $PIDFILE`
            cd $DAEMON_PATH
        if [ -f $PIDFILE ]; then
            kill -HUP $PID
            printf "%s\n" "Ok"
            rm -f $PIDFILE
        else
            printf "%s\n" "pidfile not found"
        fi
;;

restart)
        $0 stop
        $0 start
;;

*)
        echo "Usage: $0 {status|start|stop|restart}"
        exit 1
esac

I run elastalert from the command line like this /elastalert/elastalert/python -m elastalert --verbose --rule rules_folder/frequency.yaml --config config.yaml --config config.yaml

Thanks in advance!

[JimKeating2]

Here's how I had to tweak the file to start on Debian Linux:
(Thanks everyone for your suggestions!)

[Unit]
Description=Elastalert
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config /opt/elastalert/config.yaml
StandardOutput=syslog
StandardError=syslog
KillSignal=SIGKILL
PIDFile=/var/run/elastalert.pid

[Install]
WantedBy=multi-user.target

[sathishdsgithub]

@JimKeating2 @Qmando @elvarb

I followed the above steps to start elastalert as service but I get the below error message. Please assist.

● elastalert.service - Elastalert
   Loaded: loaded (/lib/systemd/system/elastalert.service; linked; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2017-07-10 10:45:17 UTC; 2s ago
  Process: 7933 ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config **/opt/elastalert/config.yaml (code=exited, status=1/FAILURE)
 Main PID: 7933 (code=exited, status=1/FAILURE)**

Jul 10 10:45:17 infosec.novalocal python[7933]: File "/opt/elastalert/elastalert/elastalert.py", line 1775, in main
Jul 10 10:45:17 infosec.novalocal python[7933]: client = ElastAlerter(args)
Jul 10 10:45:17 infosec.novalocal python[7933]: File "/opt/elastalert/elastalert/elastalert.py", line 121, in __init__
Jul 10 10:45:17 infosec.novalocal python[7933]: self.conf = load_rules(self.args)
Jul 10 10:45:17 infosec.novalocal python[7933]: File "elastalert/config.py", line 453, in load_rules
**Jul 10 10:45:17 infosec.novalocal python[7933]: raise EAException('Error loading file %s: %s' % (rule_file, e))**
**Jul 10 10:45:17 infosec.novalocal python[7933]: elastalert.util.EAException: Error loading file example_rules/example_new_term.yaml: Error initializing r...**
Jul 10 10:45:17 infosec.novalocal systemd[1]: elastalert.service: main process exited, code=exited, status=1/FAILURE
Jul 10 10:45:17 infosec.novalocal systemd[1]: Unit elastalert.service entered failed state.
Jul 10 10:45:17 infosec.novalocal systemd[1]: elastalert.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

elastalert service 
[Unit]
Description=Elastalert
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config /opt/elastalert/config.yaml
StandardOutput=syslog
StandardError=syslog
KillSignal=SIGKILL
PIDFile=/var/run/elastalert.pid

[Install]
WantedBy=multi-user.target

[JimKeating2]

Omando, I'd say first of all check the paths to make sure every path is correct. Also, the service will not start if one of the rules is not configured correctly. It looks like it is one of the rules that is stopping the service from running correctly. (Rule in the "example_rules" folder) Does it run from the command line correctly? I suggest running Elastalert from the command line to check your rules...especially at first.
This is a good command for testing your rules: root@ElastAlert:/opt/elastalert# elastalert-test-rule --count-only rule_testing/event_id_13.yaml
Best Wishes! JK

[Qmando]

@sathishdsgithub
You are using the example rules, unmodified?
If you just run elastalert manually, you will see the full error
In your output, you can only see
Error loading file example_rules/example_new_term.yaml: Error initializing r...

[JimKeating2]

I created a new folder for rules and configured Elastalert to only look in the new folder for any rules. I believe the location is set in the ElastAlert.yaml config file.

```

This is the folder that contains the rule yaml files

Any .yaml file will be loaded as a rule

rules_folder: my_rules

The rules have to be just right or you will get an error every time.
We created are using Winlogbeat on Elasticsearch for our Windows Server logs.  Here's an example of a rule that looks for failed password changes:

Alert when the rate of events exceeds a threshold

(Optional)

Elasticsearch host

es_host: host1.mydomain.org

(Optional)

Elasticsearch port

es_port: 9200

(OptionaL) Connect with SSL to Elasticsearch

use_ssl: False

run_every:
seconds: 40

aggregation:
seconds: 40

realert:
seconds: 0

(Optional) basic-auth username and password for Elasticsearch

es_username: someusername

es_password: somepassword

(Required)

Rule name, must be unique

name: Failed_Password_Change

(Required)

Type of alert.

the frequency rule type alerts when num_events events occur with timeframe time

type: any

(Required)

Index to search, wildcard supported

index: winlogbeat-*

(Required, frequency specific)

Alert when this many documents matching the query occur within a timeframe

num_events: 1

(Required, frequency specific)

num_events must occur within this amount of time to trigger an alert

timeframe:

minutes: 2

(Required)

A list of Elasticsearch filters used for find events

These filters are joined with AND and nested in a filtered query

For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

filter:

• term:
  event_id: "4723"
• term:
  keywords: "Audit Failure"

(Required)

The alert is use when a match is found

alert:

• "email"

from_addr: "[email protected]"

(required, email specific)

a list of email addresses to send alerts to

email:

• "[email protected]"
  ```

Best Wishes...JK

[sathishdsgithub]

@JimKeating2 @Qmando I fix the error by modifying the correct folder path. However, I'm unable to make the service to run during startup. I get failed to execute the operation. Please assist.

[root@infosec ~]# systemctl enable elastalert.service
**Failed to execute operation: Too many levels of symbolic links**

I followed the below procedure

more /etc/systemd/system/elastalert.service

[Unit]
Description=Elastalert
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config /opt/elastalert/config.yaml
StandardOutput=syslog
StandardError=syslog
KillSignal=SIGKILL
PIDFile=/var/run/elastalert.pid

[Install]
WantedBy=multi-user.target

`ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service`

[JimKeating2]

I would remove the service as it looks as if you have multiple service links:

Here's a good link to show you how:
https://superuser.com/questions/513159/how-to-remove-systemd-services
(Find and remove all files: find / -name elastalert.service)

Recreate service file in /lib/systemd/system/elastalert.service

Create Link:
ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service

Reload Daemon:
systemctl daemon-reload

Enable Service and Start Service:
systemctl enable elastalert.service
systemctl start elastalert.service
systemctl status elastalert.service

[flippakitten]

I found zdaemon 4.2.0 to be a pretty good solution and easy.

pip install zdaemon

Create a config file with (ie. zdaemon_conf):

<runner>
  program python -m elastalert.elastalert --conf /path_to_config/config.yaml
  socket-name /tmp/elastalert.zdsock
  forever true
</runner>

then run:

zdaemon -C /path_to_config/zdaemon_conf start

[sathishdsgithub]

@flippakitten

Let me try and update you. Can we specify the zdaemon config file in /etc path ?

[flippakitten]

@sathishdsgithub Not totally sure I understand what you mean but you can keep the zdaemon config file anywhere as long as you have permission to it. (sorry it's late)

[sathishdsgithub]

@flippakitten

How do I enable the zdaemon to start during boot ?

pip install zdaemon

<runner>
  program python -m elastalert.elastalert --conf /path_to_config/config.yaml
  socket-name /tmp/elastalert.zdsock
  forever true
</runner>

zdaemon -C /path_to_config/zdaemon_conf start

[flippakitten]

@sathishdsgithub I haven't implemented it to start at boot.

It would depend on the use case but I guess I would create a simple bash script and then call it from a crontab on reboot. This comes with other possibly issues but if the goal is to get it working for now, it should do the trick.

Something like:

create a file ie. 'start_elastalert.s'h and add

#!/bin/sh
zdaemon -C /path_to_config/zdaemon_conf start

Then

$ chmod +x start_elastalert.sh
$ crontab -e

#add cron
@reboot  /path/to/script/start_elastalert.sh