Influxdb: Cluster discovery mechanism for AWS EC2 deployments

[marco-hoyer]

I recently tested an elasticsearch cluster deployment on AWS using their awesome plugin https://github.com/elasticsearch/elasticsearch-cloud-aws which gathers cluster-members from ec2 api by searching for definable security groups and/or tags. In my opinion this is a quite smart discovery mechanism for the ec2 world and I would love to have it for influxdb, too. (If there isn't anything already?!)

[pauldix]

This seems like it's something more suited to some service discovery system, plus configuration system like Chef, Puppet, or Ansible.

[marco-hoyer]

Might be but a integrated solution for service discovery feels much nicer than something trying to instrument the application from the outside. Please correct me if I missed something but if I use some sort of service discovery for this, I have to ensure that the application gets rid of changes in the config files. This means to restart or reload the service. But since I don't want my service to interrupt, I would also need to make sure the restart to happen in a serialized manner. Much of complexity. Instead of this creating a Auto Scaling Group with the tag influxdb_cluster=my-production-metrics-cluster could be enough to do to get a scalable and self-healing cluster. Each instance would gather all instances with an equal tag and try to communicate with it. This could happen in a frequent manner to ensure existing instances get rid of changes.

[geowa4]

I'd like to see this, too. I run InfluxDB as a Systemd unit within an auto-scaling group of CoreOS instances. That unit is specified in the cloud config file. Since they're created with a launch configuration, I can't really specify the seed servers. The best I can do is fire Ansible when a new instance comes online so I can rewrite the unit file to list the other servers in the group. This is not ideal. I'd much rather have InfluxDB find the other nodes itself via the EC2 API.

[lesaux]

Hello, I've though of using consul-template to manage the seed-servers line in config.toml. It probably would make sense to manage the replication and split parameters too. Scaling-up would probably be safe for the data, but I need to figure out what would happen to data if I were to scale back down the number influxdb instances.

[pcn]

It seems like there is a opportunity to provide a way into influxdb so that the ec2 ecosystem can be used to orchestrate it.

Off the top of my head, how about a set of configurable endpoints that can be hit in order to identify membership, tagging etc? E.g.

[discovery]
# a GET endpoint that returns json with keys and values including seeds and some other params i
group_group = http://192.168.33.45/get_influx_group
# a POST endpoint that passes params including the above group name
# and this hosts identifying info like EC2 data, etc.
register_with_group = http://192.168.33.45/register_with_influx_group

This is off the top of my head, but it seems like having a few well-defined endpoints that can be hit for GETs and POSTs of this sort of thing is much more practical in that any site can develop their own methods of discovering and publishing their data, while keeping the amount of code to do this in influxdb pretty small.

[marco-hoyer]

I believe that making influxdb easy to operate on AWS is somehow a huge advantage and will make a lot of people use it. So maybe it could be a good idea to have a plugin system where one can configure a provider plugin for a list of cluster members. There could be an ec2 plugin and maybe one for etcd or similar tools.

[geowa4]

I've since started using Kubernetes, which handles all the networking for me. This is no longer a priority for me, but I can see it still being a nicety for others.

[jsternberg]

We no longer support clustering in the open source version.