Chronograf: Support automated deployment

[mleonhard]

Hi InfluxData Team,
We are using Terraform to deploy Docker containers with servers. Our continuous integration system deploys to a QA cluster and runs automated tests. We can manually start deployments to staging and production clusters. Each engineer has their own cluster for development. We need all of these deployments to draw their configuration from the source code repository. This is easy to accomplish when each deployed server reads its configuration from a file, env vars, or command args.

Unfortunately, Chronograf seems to provide only an API for configuration. This is a severe limitation for us adopting Chronograf.

This comment from a discussion thread sums it up: "Lacking the ability to do automated setup without going through the GUI is an automatic no-go for my project."

Please add a way to provide files to Chronograf that automatically create and update data sources, organizations, users, permissions, dashboards, alerts, and saved log queries.

Also update the docs to provide the knowledge required by everyone setting up Chronograf in a modern cloud-based deployment:

1. How to disable the "All new users are SuperAdmins" and all other insecure-by-default settings that all users must change. Or even better, provide a secure-by-default option and make it the default. (Remove the foot-guns.)
2. What kind of data Chronograf stores on disk
3. How much data is stored (size and number of files)
4. How frequently does Chronograf update the files (disk bandwidth requirements)
5. What happens when Chronograf is stopped, the data files deleted, and then restarted? This happens when an immutable container is updated and the data files are not stored on a persistent volume. Put another way: Which Chronograf features require a persistent data volume?
6. Does Chronograf create any missing files and directories inside its data directory? If not, then I will need to create and maintain a deployment script that does that work.
7. How to make Chronograf log to stdout and not write a logfile. Logging is usually handled outside the container.
8. Describe the format of log lines and provide complete example configs for Telegraf (with docker_logs) and other popular log parsing tools.
9. Explain memory requirements and recommend memory and CPU allocations for common deployment sizes. Please don't make us guess and perform trial-and-error.
10. Are there any special restrictions on file paths? For example, Telegraf has an undocumented requirement that certificates are stored under the config directory. This makes it harder to use secrets on tmpfs. Again, please don't make us perform trial-and-error.

Sincerely,
Michael

[glinton]

Unfortunately, Chronograf seems to provide only an API for configuration.

I'm not sure I follow, are you referring to only providing an API for creating dashboards, sources, etc... or actual chronograf configuration?
chronograf --help produces all the cli flags and corresponding environment variables for configuring a chronograf.

Please add a way to provide files to Chronograf that automatically create and update data sources, organizations, users, permissions, dashboards, alerts, and saved log queries.

Have you checked out the --resources-path config option? Documentation surrounding this and the other --*-path options can likely be improved, but you can place files in that location that contain pre-configured chronograf resources. These resources can include dashboards, sources, kapacitors, and organizations.
To create a dashboard, you can define it in json format (iirc, it's the same/similar format as fetching a dashboard from the api) and store it with a .dashboard extension. Similarly, save a source, kapacitor and org with .src, .kap, .org extensions, respectively.

This comment from a discussion thread

The discussion thread you linked is specific to InfluxDB 2.0 line, which Chronograf is not part of and thus is not subject to those unimplemented-at-the-time features referenced.

1. To disable new users being added as a superadmin, send a PUT to /chronograf/v1/config/auth with a payload of {"superAdminNewUsers":false}.
2. Chronograf stores resource data on disk (in a boltdb file or recently an etcd store) - source, dashboard, kapacitor, organization, some user definitions.
3. Depends on the extensiveness of your defined resources - particularly dashboards. For example, hit /chronograf/v1/dashboards to see how much data is being stored on disk for dashboards.
4. User updates are stored.
5. If Chronograf is stopped, and the boltDB file is deleted and there are no configured resource/canned files in place, a user would be walked through adding a source, dashboard, and kapacitor as if it was a fresh install.
6. Chronograf does not create missing files/directories at this time, feel free to open a new feature request for that in this repo though.
7. Chronograf logs to stderr by default. Currently there is no way to change this, but again, a feature request would be welcomed.
8. I can maybe see the addition of an example telegraf config to parse the current Chronograf logs, but don't think examples for "other popular log parsing tools" is anything we'd rush into. A feature request for it wouldn't hurt though; perhaps for the addition to set the output format (the logging library chronograf is using supports jsonFormatting, which could simplify machine log injestion).
   Current log format:

INFO[0000] Serving chronograf at http://[::]:8888        component=server
INFO[0003] Response: OK                                  component=server method=GET remote_addr="[::1]:43322" response_time="270.791µs" status=200
ERRO[6147] Error message Unparsable JSON                 component=server http_status =400

1. I can't say, it seems it heavily depends on the complexity of your dashboards and the usage you expect.
2. Not that I'm aware of; though I'm also not familiar with any restrictions in telegraf pertaining to certs.. Can you elaborate on that, perhaps open an issue in the telegraf repo so that can be resolved if it is found to be an issue.

thanks.
let me know if you still have questions

[kramik1]

I just spent hours trying to figure out why resources path files were not being detected. The documentation says there is a default but that doesn't appear to be the case. I had to set it in the environment variable first. I am using docker-compose at the moment. Putting this here in the interim until the documentation or the image is fixed to save people some time :)

[sranka]

Chronograf can be started with several environment variables that can pre-define some entities (dashboards, canned files, source influx ...), but unfortunately not all types are supported (users, permissions ...). The hand-made UI configuration tasks are stored in an internal bolt database file (/var/lib/chronograf/chronograf-v1.db). /var/lib/chronograf/ is a defined docker volume that can be mapped to a pre-configured directory with chronograf-v1.db file. This database file can be created in advance to store all the required configuration aspects. It can use relative addresses (hostnames) so that the repetitive deployment works fine. Hostnames is IMHO the only thing that changes. There is no better solution on the roadmap.