St2: Feature Request: `LockingPollingSensor` to make HA aware polling sensors

[djhaskin987]

Copying @armab on this.

SUMMARY

As discussed here, as a feature request, please create a LockingSensor and LockingPollingSensor class that I would be able to override much like we are presently able to do with the Sensor and PollingSensor classes.

Both sensor classes would start up, obtain a lock from the coordination backend, and continue to renew the lock (in perhaps the case of the pollinglockingsensor) using the tooz library, as it already does for other non-sensor things.

BUT WHY

When I was creating my StackStorm high availability instance, I wanted to do an Active/Active setup. In order to set this up, I set up a three-node mongodb cluster with fourth node as arbiter, a three-node zookeeper coordination cluster, and a three-node rabbitmq cluster. I chose to run every service -- st2web, st2api, st2sensorcontainer, etc. -- on both nodes behind the usual nginx reverse proxy. SSL termination and load balancing is taken care of by a load balancer in front of both nodes.

High availability totally works in an active/active set up like this for our use case, except the problem of making sensors active/active HA-ready. Sensor partitioning would not work, because the setup is active/active and I need webhooks to work in our environment. Say we have nodes A and B. If I used partioning, some of the webhooks would run on A and some on B. However, in an active/active setup, if a webhook call came in for a webhook which was running on A, but the request was actually routed by the load balancer to B, then the webhook wouldn't work. This is a sort of deal-breaker in my environment.

The solution to this problem in our environment was to simply run all sensors from both nodes. The webhooks would then work as intended; however, the PollingSensors that we are writing will need extra work to make them HA-aware. We will have to override Sensor instead of PollingSensor, and basically write a PollingSensor class by hand, except we will also have to incorporate some sort of lock to make sure only one sensor has the right to conduct polls at any given time. The feature that this ticket describes would really help make better HA-aware sensors.

ISSUE TYPE

• Feature Idea

STACKSTORM VERSION

Paste the output of st2 --version:

[root@my-ha-host-01 ~]# st2 --version
st2 2.8.1, on Python 2.7.5

OS / ENVIRONMENT / INSTALL METHOD

Custom install using Chef and the StackStorm RPM repository on RHEL 7.

STEPS TO REPRODUCE

The new classes might be used like this:

The python code would be almost the exact same as for PollingSensor in the case of LockingPollingSensor:

from st2reactor.sensor.base import LockingPollingSensor

class MyHAAwareSensor(LockingPollingSensor):
    def setup(self):
        pass
    def poll(self):
        # poll is conducted here
        # LockingPollingSensor is basically a drop-in replacement for PollingSensor

The sensor config might have new config items, lease_interval and lease_duration, in addition to poll interval, which might specify how often the sensor should re-obtain the lease and how long leases should last.

---
    class_name: "HAAwareSensor"
    entry_point: "haaware_sensor.py"
    poll_interval: 600
    lease_interval: 30
    lease_duration: 30

EXPECTED RESULTS

The LockingPollingSensor would obtain a lock every lease_interval seconds. the lease would hopefully last lease_duration seconds, and would conduct a poll every poll_interval seconds.

[johnarnold]

I have the exact same issue with a polling sensor for our ticketing system. I'm looking at rolling my own locks (either via redis or the k/v backend) but it would be nicer to have locking baked into a sensor class.

[djhaskin987]

If it helps @johnarnold , we rolled our own lock as well. We ended up configuring a stackstorm backend coordination service using Zookeeper. Since we had to do this, we re-used our zookeeper cluster for the locking in the polling sensor. Working great so far :)

[igorfernandes]

+1

This feature would be great for us too.

@djhaskin987 can u share your LockingPolling code?

[djhaskin987]

I can't share all of it, company property, that sort of thing. However, this is the the basic outline:

#!/usr/bin/python

from st2reactor.sensor.base import PollingSensor
import kazoo.client
import socket
import os
import datetime

__all__ = [
    'MyLeasingPollingSensor'
]

class MyLeasingPollingSensor(PollingSensor):
    def __init__(self):
        self.logger = self.sensor_service.get_logger(name=str(self.__class__.__name__))
        self.zookeeper_instance = self._config['zookeeper_instance']
        self.zk_path = self._config['zk_path']
        self.zk_lease_id = '/'.join([str(self.__class__.__name__),
                                        socket.getfqdn(),
                                        str(os.getpid())])
        self.lease_interval = int(self._poll_interval * 1.5)
        self.zookeeper_instance = self._config['zookeeper_instance']
        self.zk = kazoo.client.KazooClient(hosts=self.zookeeper_instance)

    def acquire_lease(self):
        try:
            self.logger.info("Attempting to acquire lease `{lease}`...".format(lease=self.zk_path))
            self.zk.start()
            self.zk.ensure_path(self.zk_path)
            lease = self.zk.NonBlockingLease(self.zk_path,
                    datetime.timedelta(seconds = self.lease_interval),
                    identifier = self.zk_lease_id)
            self.zk.stop()
            if not bool(lease):
                self.logger.info("Failed to acquire lease `{lease}`".format(lease=self.zk_path))
                return False
            self.logger.info(
                    "Succeeded in acquiring lease `{lease}` for {interval} seconds".format(
                        lease=self.zk_path,
                        interval=self.lease_interval))
        except Exception as e:
            self.logger.error("Error encountered while acquiring lease: {e}".format(e=str(e)))
            return False
        return True


    def poll(self):
        self.logger.info("My Sensor now polling")
        if not self.acquire_lease():
            return
        # ... The rest of the poll() method ...

    # boilerplate
    def cleanup(self):
        pass

    # boilerplate
    def add_trigger(self, trigger):
        pass

    # boilerplate
    def update_trigger(self, trigger):
        pass

    # boilerplate
    def remove_trigger(self, trigger):
        pass

Its YAML file might look like this:

---
  class_name: "MyLeasingPollingSensor"
  entry_point: "my_leasing_polling_sensor.py"
  description: "Sensor to poll stuff on an HA StackStorm instance."
  poll_interval: 600
  trigger_types:
    - 
      name: "my_trigger"
      description: "Trigger for my stuff."

The config.schema.yaml for the pack in which it resides might look like this:

---
  zk_path:
    description: "ZNode path to the lease information for the poller"
    type: "string"
    required: true
  zookeeper_instance:
    description: "Instance of ZooKeeper to use"
    type: "string"
    required: true

An example config file for the pack:

zookeeper_instance: "zookeeper-server-1.example.com:2181,zookeeper-server-2.example.com:2181,zookeeper-server-3.example.com:2181"
zk_path: "/myorg/sensors/mysensor"

I create an instance of a kazoo.client.KazooClient from the python zookeeper kazoo library. I point it at the zk_path. As an identifier for the lease, I use the machine name, the OS process ID, and the class name. This should be unique enough for an HA deployment. Every time I poll, I attempt to acquire the poller's lease. I ask for a lease one and a half times longer than the poll interval. This way, whichever machine acquires the lease will prevent the other machine from polling as long as it has the lease, which it renews at every poll interval.

You'll notice I'm getting a lease here, not a lock. By getting a lease, I ensure that the poller on the other HA node can't fire triggers as long as the first node is up. I simply make sure the lease time is 1.5 times the length of the poll interval (self._poll_interval in a StackStorm standard Poller class), and then if the first HA node is up, it will always come up and renew its lease, ensuring that it will keep the lease as long as it is up. If it ever goes down, the lease will expire and the other HA node will take over.

EDIT: The reason I use zookeeper in the above code is because we use zookeeper _anyways_ as the coordinator server for the StackStorm instance.

[vivekdatar]

A polling arbitrator running on "controller box" may be a good long term and scaleable solution. Let us call it PA. This arbitrator would elect an "operational polling sensor" from its list, which will be pre-configured. The polling sensors, in HA mode, will have to register to the arbitrator, and implement a simple heartbeat.

Assumption: Actionrunner and polling sensors are combined in a given node. i.e. polling sensor from node 1 can only trigger action on node 1. In other words, polling sensor in node 1 cannot trigger an action in node 2.

The PA would determine polling-sensor failure and will elect a new operational polling-sensor from a different node. There are multiple levels of safety-nets to prevent multiple polling-sensors to start polling.

1. Polling sensors must register with PA, and will be "told" to start polling
2. Polling sensors will cross-verify their status after a period of time
3. PA will tell polling-sensor to start polling and at the same time inform st2actionrunner if the polling agent is operational on that node or not.

So the code changes will be in following areas.

• Arbitrator will be new development
• Polling sensor behavior changes in HA mode
• st2actionrunner behavior changes in HA mode

Clearly a design doc has to be written, which I am willing to do.

There is a possibility that majority of this work can be done via Sentinel. I am open to suggestions.

[nmaludy]

@vivekdatar wonder if you could accomplish something very similar with Consul locking? https://www.consul.io/docs/commands/lock.html

Could use consul lock to run your sensor, or have your sensor run on every node and call the Consul lock API in your code around the part where you do "real" work.

[armab]

I just moved to open community a couple of Discussion threads we had before about Sensors HA. This will give more context and food for thought.

See:

• What does it take to make StackStorm Available and Scalable?
• HA - Banter

[vivekdatar]

Polling Sensor HA Design Document.docx

Initial version. Comments welcome. Thx

[vivekdatar]

All: Is it sufficient to post a document here, or should I check it into github for people to comment?

[armab]

@vivekdatar Please create a new dedicated Github Issue here: https://github.com/stackstorm/discussions/issues with your proposal/design. There are other examples of previous design discussions in that repository.

Don't forget to post the text/contents of the document instead of attaching the file itself.

Thanks!