Kapacitor: [Feature Request] Kapacitor needs a way to automatically load tick scripts from a directory.

[james-lawrence]

having to manually invoke kapacitor for each script is pretty annoying for deployment. We should just be able to load from a directory. Main goal is to put the scripts under version control and ease of deployment.

things that may need to be thought about:

how does kapacitor know which db/rp to use?

• could implement a directory structure. scripts/{dp}/{rp}/myscript.tick

how could templates be handled?

• not sure havent used these yet.

[sputnik13]

why not write a client side script that does this? the script can iterate through whatever directory structure you have and invoke kapacitor client each time there's a change

[james-lawrence]

who says I didn't do exactly that. the problem was it was 1 additional thing I had to think about to make kapacitor operational within the infrastructure.

I mentally made a note thinking 'this is retarded, these are config files why do I need to write scripts to get a basic instance up' it felt like additional unnecessary work that should be builtin to kapacitor.

[nathanielc]

@james-lawrence I agree this would be a nice additional feature. How we do this internally is similar to what @sputnik13 suggested by using a script. But instead of a directory structure we have special comments in the TICKscripts that contain the dbrp information.

We have bounced around the idea of creating a standard way of specifying the dbrp in the TICKscript to make this easier.

Thanks for filing this feature request, like I said it is something we have discussed before and it good to see others find value in it too.

[sofuture]

Also agree this is a very helpful feature.

[goller]

I like the idea of comments specifying things like db and rp. What about going further and making a special comment that is a command to run similar to #! for shell scripts or //go:generate for go?

Maybe like this:

// kapacitor:define my_tick -template my_tmpl -dbrp mydb.myrp -vars my_vars.json

[nathanielc]

@goller I like emulating the special go comments but I was thinking along different lines. More like build tags than the go:generate comments. https://golang.org/pkg/go/build/#hdr-Build_Constraints

For example your line would become:

// template:my_tmpl
// dbrp: mydb.myrp
// vars: my_vars.json

But the above doesn't really make sense because if the task being defined is sourced from a template then it doesn't have a .tick file in which to place the comments.

Are you thinking that the template file would have a comment per task created from the template?

So an example that isn't a template would look like this:

// dbrp: mydb.myrp
// dbrp: anotherdb.anotherrp

Two other things struck me as odd:

• Your kapacitor:define example implies that the kapacitor CLI would be executed to actually define the task which may not be the case. But is does allow a quick bash script to be written and find the comments and run them via the CLI. So no sure which I like best.
• I would like the filename to be used as much as possible. In the kapacitor:define example the name of the task is explicitly specified, where I would prefer it to be sourced from the file name containing the script.

Just some thoughts....

[devth]

This feature would help when setting up TICK with automation (as opposed to by hand).

@james-lawrence care to share your script that loads your TICKs?

[jackzampolin]

@devth You can see an example of a loader script in the promtheus_metric_normalizer.

[devth]

I added a post-upgrade,post-install Job to the Helm Chart to load TICK scripts from a directory. Looks like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "fullname" . }}-init
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
data:
  init.sh: |-
    #!/usr/bin/env bash
    export KAPACITOR_URL=http://kapacitor:9092
    ls -al $TICK_DIR
    kapacitor list tasks

    echo "Loading TICK files from $TICK_DIR"
    for f in ${TICK_DIR}/*.tick
    do
        name="$(basename $f)"
        name="${name/.tick/}"
        kapacitor define $name \
            -type batch \
            -dbrp telegraf.autogen \
            -tick $f
        kapacitor enable $name
    done
    echo "DONE"
    kapacitor list tasks
---
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ template "fullname" . }}-init-job
  labels:
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: {{ .Release.Name | quote }}
    heritage: {{ .Release.Service | quote }}
    app: {{ template "fullname" . }}
  annotations:
    "helm.sh/hook": post-install,post-upgrade
spec:
  template:
    metadata:
      name: "{{.Release.Name}}"
      labels:
        heritage: {{.Release.Service | quote }}
        release: {{.Release.Name | quote }}
        chart: "{{.Chart.Name}}-{{.Chart.Version}}"
    spec:
      restartPolicy: OnFailure
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        command: ["bash", "/init/init.sh"]
        env:
          - name: TICK_DIR
            value: "/tick"
        volumeMounts:
        - name: alerts
          mountPath: /tick
        - name: init
          mountPath: /init
      volumes:
      - name: alerts
        configMap:
          name: {{ template "fullname" . }}-alerts
      - name: init
        configMap:
          name: {{ template "fullname" . }}-init

Storing the script in a ConfigMap lets me avoid having to bake/push my own Docker image. In this case I'm using the official kapacitor Docker image.

[jackzampolin]

@devth Do you mind putting up a PR with that work in it?

[james-lawrence]

sorry just wrapping around to this again. here is the script I made.

#!/bin/bash
set -e

# load script files
ls kapacitor/alerts/ops | xargs -i kapacitor define "{}" -type stream -dbrp ops.autogen -tick kapacitor/alerts/ops/"{}"
ls kapacitor/alerts/app | xargs -i kapacitor define "{}" -type stream -dbrp app.autogen -tick kapacitor/alerts/app/"{}"

kapacitor list tasks | tail -n +2 | awk '{print $1}' | xargs -i kapacitor reload "{}"
kapacitor list tasks | tail -n +2 | awk '{print $1}' | xargs -i kapacitor enable "{}"

[desa]

@james-lawrence @devth @sputnik13 @sofuture @goller

We just recently implemented https://github.com/influxdata/kapacitor/pull/1481. Here is an example of how we intend it to be used. It would be great if you could give it a try and offer us any feedback.

[2solt]

@desa
I was testing with batch tick scripts, only working if I define db and rp in the batch query.
e.g. SELECT response_time FROM "test"."default"."net_response"
If Im not putting it in I get the following error (SELECT response_time FROM "net_response"):

ts=2017-11-14T16:00:44.329Z lvl=error msg="encountered error" service=run err="open server: failed to reload tasks/templates/handlers: failed to load tasks: failed to load file /etc/kapacitor/tasks/test.tick: failed to create task: batch query is not allowed to request data from \"\".\"\""
run: open server: failed to reload tasks/templates/handlers: failed to load tasks: failed to load file /etc/kapacitor/tasks/test.tick: failed to create task: batch query is not allowed to request data from "".""

Defining database and retention policy in the batch query is still a requirement even if dbrp is defined in the beginning of the tick script? (dbrp "test"."default")

Any other scenario woks correctly.

[desa]

Defining database and retention policy in the batch query is still a requirement even if dbrp is defined in the beginning of the tick script? (dbrp "test"."default")

It is still required in the query. Specifically for the situation like the following

dbrp "telegaf"."autogen"
dbrp "telegaf"."downsampled"

batch
  |query('SELECT * FROM "telegraf"."autogen"."cpu"')
    ...

batch
  |query('SELECT * FROM "telegraf"."downsampled"."cpu"')
    ...

[cabrinha]

@desa autoloading tick scripts via tasks doesn't seem to be working using kapacitor:latest docker image.

env | grep KAPACITOR
...
KAPACITOR_LOAD_DIR=/var/lib/kapacitor/load
KAPACITOR_LOAD_ENABLED=true
...
# ls /var/lib/kapacitor/load/tasks/
test.tick
# kapacitor list tasks
ID Type      Status    Executing Databases and Retention Policies

[desa]

@internaught This feature is available in 1.4 (which is currently unreleased, but available a a release candidate) and I think the latest docker image still points to 1.3.

[cabrinha]

@desa is there any way I can grab the 1.4 RC and build a docker image from it?

[desa]

@internaught yeah

https://dl.influxdata.com/kapacitor/releases/kapacitor-1.4.0~rc3-static_linux_amd64.tar.gz
https://dl.influxdata.com/kapacitor/releases/kapacitor_1.4.0~rc3_amd64.deb
https://dl.influxdata.com/kapacitor/releases/kapacitor-1.4.0~rc3.x86_64.rpm

Here are the packages. Sorry they're not listed anywhere else.

[cabrinha]

@desa thanks a lot, working great now! side note that https://dl.influxdata.com/kapacitor/releases/kapacitor-1.4.0~rc3-static_linux_amd64.tar.gz.asc does not exist, so I had to remove some of the GPG verification lines from the Dockerfile

[roch1990]

Tick script loading - doesnt work. :(
I'm using kapacitor 1.4.0 into docker container (with influx).
Kapacitor from:
https://dl.influxdata.com/kapacitor/releases/kapacitor-1.4.0.x86_64.rpm
Tick script dir from conf:
dir = "/etc/kapacitor/load"
In addition - env:
KAPACITOR_LOAD_DIR=/etc/kapacitor/load
Tick script directory rights - 755.

# kapacitor list tasks
ID Type      Status    Executing Databases and Retention Policies

What is wrong?

[desa]

@roch1990 are you getting any errors?

[roch1990]

@desa Nope, kapacitor started without errors.
Docker container - based at CentOs7 image.
System manager - s6-overlay.

[desa]

@roch1990 could you open an question on https://community.influxdata.com/. This way the larger community can benefit from the answer.

Also please report the link back here so I can respond to it.

[roch1990]

@desa
Done

[petemyron]

just to follow this up--the issue above where telegraf started ok without loading the tasks from disk was due to the conf file having the wrong directory structure (ask how I know).

In the conf, you should have:

[load]
  enabled = true
  # this dir requires there to be a __subdirectory__ called 'tasks'
  dir = "/var/lib/kapacitor/"

And then your .tick files should live in /var/lib/kapacitor or wherever you specify. See here

[james-lawrence]

just gave this a spin one question: how do I disable one of the tasks from the tick script file?

[desa]

@james-lawrence Great question. Would you mind asking it https://community.influxdata.com/ so that the larger community could benefit from the answer? 👍

[desa]

Closing issue since feature has been delivered. Future improvements should open new issues/feature-requests

[sramshaw-driver]

I must be overlooking something, but I simply do not understand why you can specify a dbrp in a template or taskfile for stream tasks but batch tasks won't use it and you have to hard code it. This seems like a massive shortcoming for batch tasks.
Even in this example https://docs.influxdata.com/kapacitor/v1.4/guides/continuous_queries/, doesn't it seem not-ideal that "telegraf"."autogen" has to be statically duplicated and hard coded into the query when it is already specified as the dbrp?

[desa]

@sramshaw-driver one tickscript can query more than one dbrp.

dbrp "telegraf"."autogen"
dbrp "other"."autogen"

var x = batch
    |query('SELECT mean(usage_idle) as usage_idle FROM "telegraf"."autogen".cpu')
...

var y = batch
    |query('SELECT mean(usage_idle) as usage_idle FROM "other"."autogen".example')
...