Splunk-connect-for-kubernetes: Metrics are not going to specified metrics index

[MarkPare]

What happened:
metrics are not going to metrics index (my-metrics-index) in splunk.

What you expected to happen:
metrics logging goes to metrics index.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
Confirmed that my metrics index my-metrics-index is of type metric.

Environment:

• Kubernetes version (use kubectl version):
  1.11.0
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release):
• Splunk version:
  7.0
• Others:

values.yaml contents:

global:
  logLevel: info
  splunk:
    hec:
      token: my-token
      host: my-host.com
      port: 8088
      protocol: http
  kubernetes:
    clusterName: "my-cluster"

splunk-kubernetes-logging:
  splunk:
    hec:
      indexName: my-logging-index

splunk-kubernetes-objects:
  splunk:
    hec:
      indexName: my-objects-index

splunk-kubernetes-metrics:
  splunk:
    hec:
      indexName: my-metrics-index

[matthewmodestino]

Hi @MarkPare!

How are you deploying? Helm? Manifests?

We need to check the rendered configs. One good way to do that is to get the first 1000 lines of your metrics collector pod, as we dump the config there.

kubectl -n splunk logs -f <yourMetricsPod>

Or check in Splunk for your pod name. (search like this | mcatalog values(metric_name) WHERE index=<yourindex> by host if you have not installed app for infra or a fields.conf)

If you can share that I can usually spot the necessary actions needed.

Also, the HEC token and Splunk side config can play a role in metrics final resting place. But we can look at that after we validate the metrics pod hec config.

[chaitanyaphalak]

@MarkPare closing issue due to inactivity. Please reopen/create a new one if you need further support.

[MarkPare]

@matthewmodestino thanks for your help on this and sorry for not replying sooner. The issue had to do with my noob-ness using splunk interface. The logging was working correctly.

Thanks to all contributors on this project for awesome work. This library is awesome!