Chronograf: Datasource should be persisted if created from the command line

[nhaugo]

This may be easier accomplished via api calls @lukevmorris

[timraymond]

I think this might be introducing surprising behavior. If I start Chronograf with:

chronograf --influxdb-url=http://localhost:8086

I would first expect that to be idempotent, which is easy enough to solve by making sure hostnames and ports are unique within the source bucket. However what should the behavior be when it's started with a different URL? Should it also persist that? Which one should be the default?

I think it might make more sense to migrate to a command structure like:

chronograf add-source https://localhost:8086 --default --insecure-skip-verify --username=<username> --password=<password>

This would exit immediately but update the Bolt store. The server than could be started with chronograf serve.

[nhaugo]

Lets do a combo of these ideas. A flag --add-source that takes a json string or something else that contains:
the id
the name
the url
the user
the pass
whatever else is needed for a data source

This will try to create a new data source if it does not exist, if the id exists it does nothing with a successful return.

[jaredscheib]

RFC:

My plan is to start with the following, where we pass this new flag a stringified JSON value with InfluxDb and Kapacitor connection information, e.g.:

--new-source="{"influx":{"id":"0","name":"Influx 1","username":"user1","password":"pass1","url":"http://localhost:8086","metaUrl":"http://metaurl.com","default":true,"telegraf":"telegraf","links":{"self":"/chronograf/v1/sources/0","kapacitors":"/chronograf/v1/sources/0/kapacitors","proxy":"/chronograf/v1/sources/0/proxy","write":"/chronograf/v1/sources/0/write","queries":"/chronograf/v1/sources/0/queries","permissions":"/chronograf/v1/sources/0/permissions","users":"/chronograf/v1/sources/0/users","roles":"/chronograf/v1/sources/0/roles"}},"kapacitor":{"id":"0","name":"Kapa 1","url":"http://localhost:9092","active":true,"links":{"proxy":"/chronograf/v1/sources/0/kapacitors/0/proxy","self":"/chronograf/v1/sources/0/kapacitors/0","rules":"/chronograf/v1/sources/0/kapacitors/0/rules"}}}"

The above value is the stringified form of:

{
  "influx":{
    "id": "0",
    "name": "Influx 1",
    "username": "user1",
    "password": "pass1",
    "url": "http://localhost:8086",
    "metaUrl": "http://metaurl.com",
    "default": true,
    "telegraf": "telegraf",
    "links": {
      "self": "/chronograf/v1/sources/0",
      "kapacitors": "/chronograf/v1/sources/0/kapacitors",
      "proxy": "/chronograf/v1/sources/0/proxy",
      "write": "/chronograf/v1/sources/0/write",
      "queries": "/chronograf/v1/sources/0/queries",
      "permissions": "/chronograf/v1/sources/0/permissions",
      "users": "/chronograf/v1/sources/0/users",
      "roles": "/chronograf/v1/sources/0/roles"
    }
  },
  "kapacitor": {
    "id": "0",
    "name": "Kapa 1",
    "url": "http://localhost:9092",
    "active": true,
    "links": {
      "proxy": "/chronograf/v1/sources/0/kapacitors/0/proxy",
      "self": "/chronograf/v1/sources/0/kapacitors/0",
      "rules": "/chronograf/v1/sources/0/kapacitors/0/rules"
    }
  }
}

You should be able to pass multiple of the same flag as arguments to your Chronograf server process, i.e. 2x --new-source with two different Influx+Kapa configs.

Should --new-source be idempotent by name, or by host?

[nhaugo]

I would remove the "id" from both, and let the system decide, otherwise i could foresee sadness

[nhaugo]

Also, do we need the links or can the system figure those out.

[jaredscheib]

Sounds good on id.

I wondered the same on links, was going to ask @goller for comment.

And what of the idempotence question?

[jaredscheib]

Checked the docs again and looks like links and id are not Required.

Here's our current draft that we're working from:

[
  {
    influxdb: {
      name: "Influx 1",
      username: "user1",
      password: "pass1",
      url: "http://localhost:8086",
      metaUrl: "http://metaurl.com",
      insecureSkipVerify: false,
      default: true,
      telegraf: "telegraf"
    },
    kapacitor: {
      name: "Kapa 1",
      url: "http://localhost:9092",
      active: true
    }
  },
  {
    influxdb: {
      name: "Influx 2",
      username: "user2",
      password: "pass2",
      url: "http://localhost:8086",
      metaUrl: "http://metaurl.com",
      insecureSkipVerify: false,
      default: false,
      telegraf: "telegraf"
    },
    kapacitor: {
      name: "Kapa 2",
      url: "http://localhost:9092",
      active: true
    }
  }
]

aka:
--new-sources='[{"influxdb":{"name":"Influx 1","username":"user1","password":"pass1","url":"http://localhost:8086","metaUrl":"http://metaurl.com","insecureSkipVerify":false,"default":true,"telegraf":"telegraf"},"kapacitor":{"name":"Kapa 1","url":"http://localhost:9092","active":true}},{"influxdb":{"name":"Influx 2","username":"user2","password":"pass2","url":"http://localhost:8086","metaUrl":"http://metaurl.com","insecureSkipVerify":false,"default":false,"telegraf":"telegraf"},"kapacitor":{"name":"Kapa 2","url":"http://localhost:9092","active":true}}]'

[jaredscheib]

As for the idempotence question, @timraymond brings up the complication of running behind ALBs: if you are running behind an ALB, your IP would not change, whereas if you are _not_, your IP may change.

This is in the context of a user adding a new source that may already exist -- what should we be checking as the unique identifier for a source?

Seeking insight from @nhaugo and @goller.

[jaredscheib]

And @nhaugo, could any of these have default values, such that the user could skip setting them via the CLI?

[jaredscheib]

@goller recommended that instead of JSON, we use the built-in package http://godoc.org/github.com/jessevdk/go-flags and specifically implementing https://github.com/jessevdk/go-flags/blob/master/examples/main.go#L21, using something like a comma-delimited map[string]string data type of key:value pairs, comma-delimited.

[nhaugo]

"Should --new-sources be idempotent by name, or by host?"

Name

[nhaugo]

Instead of json use the format --howdy one:two,three:four
Then shell escape the password

[jaredscheib]

👍

[jaredscheib]

In speaking with @timraymond, we're going to start with JSON encoding since the first use case for this is via scripts, which can easily output JSON. At the point we want to support a more human-friendly interface, we can use the custom format above, and allow for both by checking the first byte's delimiter.